Yeah we’ve discussed no longer associating chat channels with category permissions before, so it’s possible that we may do that in the future.
At the moment, as you discovered, to be able to join a chat channel you must have reply permissions in the attached category. There’s no concept of a read-only chat channel in Discourse yet.