I can see that. My case right now might be an edge case, and I already have to notify the users directly about what happened so the automatic notification is not so pressing.
Still it seems uneven to not notify about the losing of the badge. Maybe there could be an admin setting to allow admins to “Notify users when they earn this badge” and “Notify users when they lose this badge”.