I’ve had a similar forum in mind for a while and I’ve been put off starting for the same reasons.
I came up with a possible solution of having a “flagger” group but it would mean keeping a score of correct flagging and incorrect flags. You could then have “trusted” flaggers where topics or posts can be removed as if they were trust level 3 but have no admin moderation access. A weighting system would need to be worked out.
Plus possible training before allowing into group in first place.
I was worried about “MAGA” versus “Woke” type intolerance, I thought empowering some users might work.