Thanks Moin…this is very unexpected, and AFAIK you are the only person that has encountered this so far. I will make a test site for myself and make changes + deploy to see if I can replicate the issue, maybe that will help track it down.
It definitely feels like some caching issue, especially as you say the UI itself is flip/flopping, but you can see in admin that the setting is still enabled.