That is correct.
In a standard forum, I think deleted topics explain most discrepancies between the topic count displayed in the tag list and the actual number of topics a user can see within each tag. Fixing this would already be a big plus. But notice that discrepancies also occur when a non-admin user is looking at a tag containing private topics.