Does the existing max_notifications_per_user site setting help here? It is a hidden setting but it does a similar job, it limits the total number of notifications on an instance.
That setting by default will keep the last 10,000 notifications per user, and automatically purge the rest in a background job that runs every week. You could lower it to a smaller number (1000 maybe?) and then it’ll clean up even more aggressively.
Context via the original commit: https://github.com/discourse/discourse/commit/372f6f4f221e1fe3027be32ed3f22ab6b5b22d48