User notification level revert after page reload

[JammyDodger]

I can replicate this here on Meta.

Interestingly, I tried it on an older site and it works, so presumably something inbetween this version and now (508370e433).