However, if the underlying problem isn’t fixed, it will come back when you reload the page.
This is known behaviour. The current feature is to get rid of “stuck” or one-off notices. We’re working on extending this functionality to let you “snooze” these notifications for a set time frame, so they won’t show up immediately again.