He took it off without asking and then told me. What now?

Four months in, going well, we have a short list of rules and one of them is that it does not come off without me. On Saturday he took it off. Nothing dramatic, he was uncomfortable after a long day and wanted a proper wash, and he put it back on afterwards. He told me on Sunday morning, unprompted, and he was clearly miserable about it.

My honest first reaction was relief that he told me. My second reaction was that if I do nothing, the rule is not a rule. My third reaction was that I am not sure I even want to be the sort of person who does something about it.

So I have done nothing for four days and it is sitting between us. What would you do here? I do not want to overreact and I do not want to pretend it did not happen.