I believe clicking Toggle whisper again also turns it back off, are you trying this and it is not working? See a gif below where it works just fine for me.
Toggle whisper
