I have two thoughts here. Neither is a complete design, but hopefully they are helpful.
Note: the complexity here is in relationships between the types of notifications. Imagine removing all settings except these (shown in topics, sub-categories, categories, and globally):
Notifications: Inherit | All posts | First post | Mentions+replies | Never
Tracking: Inherit | All posts | First post | Mentions+replies | Never
Emails: Inherit | All posts | First post | Mentions+replies | Never
Global wouldn’t have “inherit”, since it’s the top level.
Topics wouldn’t have “first post”, since it can’t apply retroactively.
Reducing the number of settings — for example, by merging “tracking” and “notifications” into one continuum — makes notifications quicker to configure, but doesn’t necessarily make them simpler when you also have to consider their relationship to email settings.
⇒ If/when you switch to per-category mailing list mode, the simplifications / relationships you have now may not be the ones you want to stick with.
This is what brought me here. To me, “mailing list mode” (as compared to “send me emails for every notification”) means that I am taking responsibility for managing notifications (in my email) — discourse should assume that I have read every post that is emailed to me. For categories with mailing list mode on, I should never receive unread notifications (I would still appreciate receiving notifications which are automatically marked as read).