We were indeed missing some “cleanup” path when removing users from PMs (either directly or through a group).