Category Security Permissions

What is the purpose of these roles, then? Will users with these levels be able to see, reply, create even if they are not added to the Security list in a category?

Put differently, can I remove then the security of a category to declutter?