These are mutually exclusive goals.
Mailing list mode means, by definition, you want an email of every single (public) post in the system. That is its purpose, its reason for existing. All filtering has to – again, by definition – be performed on the mailing side.
It sounds like you really don’t want mailing list mode. Just have the relevant folks go into the target categories and set those categories to
- Watching (notify me of every post in every topic in this category)
- Watching First Post only (notify me of the first post in each new topic in this category)
You can also set the user preference “email me even if I am active on the site” to make sure people get “double notified”, that is, notified once in the web UI while they are active on the site, and then emailed the same notification they just got in their web browser.