I think the Trust Level is recalculated automatically once removed from a group that grants a higher one than they would have achieved normally.
However, you can use the groups you’ve created for these users within the category permissions themselves if that would be a better fit?