It is much more than that. Getting an email notification seems to completely reset the counting such that no topics older than the notification get into the summary. This has always irked me.
Basically, if the email-notified punter does not visit the site, there is zero exposure of any topics between the previous summary and the last email notification. And it makes the site look much quieter than it really is when a summary finally squeaks through.
I’d like to see it being as you suggest, where only the notified topics are suppressed in the summary for users who receive an email notification but do not visit the site. That would be a vast improvement for sites who do quite a bit of default watching!!