On a new installation, this did not happen. The user (already logged in) got a permissions error saying they didn’t have access to the topic which means when the topic belongs to a restricted category, the user was not automatically added to the allowed groups on invitation. So maybe this is a bug or its as Jeff stated “you must include group membership” and it’s not automatic.
The problem was in the context of allowing a user access to a single topic in a category while not allowing that user to view any other topics in that category. Based on Jeffs response (no per-topic permissions), that’s not possible, which is why I’m doing the work around in my previous post.