How to define custom permissions for staff, admins, moderators

That’s not possible by default since admins have unrestricted access to all the sections of the forum. Depending on what kind of permissions you want to revoke, maybe demoting them to moderator can be helpful?