Ahah! I think I’ve found the issue 
https://github.com/discourse/discourse/pull/36626/commits/97f2719231d549b7c38571bd7b8c351275e8d3fa
The @silent option wasn’t propagating properly which caused users watching the destination category to receive notifications even when the “Perform this action silently” checkbox was checked.