Hmmm, the problem is that users do get logged out, but then they get automatically logged back in immediately. One solution is to set the logout redirect setting to some external site, so that people don’t get logged straight back in. Or maybe you could even set it to /login 
I’ll have a think about this over the next few days and see if we can come up with a better solution. It is confusing to have a logout button that does… nothing.