They would appear in a section below admins and moderators with the category name denoting them as moderators for that category.
You could use CSS to hide this.
Maybe wait to see if anyone else presents a more palatable solution if this is sounding like too much change.
Learning about trust levels is always good though, I would start here and then maybe take a look at the permissions table.