If that’s the case I would assume it’s a Contribute > Bug. Just to verify, you are saying that if you make a new category, the security tab only shows trust levels 0-4, everyone, admin, moderators, and staff, but if you go to Edit a category you then see all the above plus custom groups?
Edit: Just tested and I can confirm that the security tab works as expected. All custom groups appear regardless of whether I am creating a new category or editing an existing one. So, not a bug.