I got a little sidetracked after my holiday but I’m hoping to revisit the scoring shortly and see if I can fix some of the gripes.
6 Likes
So, I finally got time to upgrade today, and this is one of our hidden posts. Can you explain why it was hidden?
Here is another example, where a low thust level user can hide a post with one flag on a system with low as hide post senstivity. How can I tune the system to require more flags before hiding stuff?
I tested with setting high as hide post sensitivity, seems it will be more likely to hide posts, can you add settings that have at least 10 or even 15 as Score to Hide Post ?
Can you try setting the sensitivity to low and see what the value is? What sensitivity was it set to?
Thanks for your feedback, it’s really useful.
I honestly cannot understand why the first example you posted happened. Is it possible you changed the sensitivity after the post the hidden? There doesn’t seem to be another way for a TL0 user to hide with a lower score than the threshold.
I might actually need to store the score to hide at the time it was hidden for future debugging.
Now, for your second question about the thresholds being off. I do agree that if the lowest sensivity is 4.8 for you and a TL0 can get a score of 6 that something seems off here.
My first question based on the first screenshot is how did a TL0 user get a score of 5 of accuracy? Are you locking users at TL0? Because they’d need at least 5 flags for their accuracy to take effect. It seems unusual for me for a TL0 user to stay at that trust level while performing such accurate flagging. That implies that 100% of their flags were agreed with.
Can you tell me how many of their flags were agreed / disagreed / total? You can see this by hovering your mouse over the thumbs up icon on the reviewable.
2 Likes
I can check, we swapped from default to low pretty recently, but I think this post was new. I was thinking perhaps accuracy has been reduced after the post was hidden for the user, if you dont save the stats from when it was hidden.
All our users are at TL0 or TL1, we are using discourse slightly different that the mainstream usecase.
So it may be possible that he flagged 5 posts without changing trust level, I have not dug deeply into how trust levels work. We also had the default settings with regards to sensitivity, so hiding posts by flagging seems to be pretty easy.
I can check later, I dont have access to the data from the location I am on.
I’d like to hear back once you’ve got a chance to review the data.
Part of me also thinks that if a user has been correct in 100% of the flags they’ve performed in the past, especially since there are more than 5, that the post should be hidden. That user knows what they’re doing!
3 Likes
Unfortunatly we get lots of user feedback that the system is broken, so apparantly that is not true, a user just needs to piggy back on a few other flags and then he has godly powers.
2 Likes
I tend to agree a single flag alone, no matter the trustworthiness or the user, should not be enough to hide a post that is not spam.
5 Likes
The user that has 5.0 as accuracy on tl0 has 100 accurate flags, so I can understand why he gets a high score, but that user does not want to hide flags with one post, he wants to notify that he thinks this is inappropritate and see if the other uses agree. At least that is true for the users that contact us an compain. I can also find other uses with 100% agree rate and only 5 flags, they have the same power, and then I assume the number of approved flags will quickly rise.
I can also see a user with 84 % agree on 37 flags, he gets accuracy of 4.2, which is enough to hide stuff with one flag on tl0 and low sensitivity.
If he hides stuff with the first flag he cannot flag slightly inappropriate stuff, but needs to learn to wait until someone is way out of line.
Our user are accustomed to the 3 strikes and you are out, so please consider making that an option again, at least while you are tuning the system. We are considering downgrading to an old version of the source if this is not possible, but are slightly worried about the data consistency of the backups when downgrading.
4 Likes
Please give us a little more time to make tweaks.
I definitely see the issue here with one user being able to hide things not being desirable, and the piggy backing on other flags. I have a couple ideas and will have something to try shortly.
8 Likes
Sounds good, if you want feedback directly, feel free to outline your ideas.
If I can get to a state where 3 to 5 flags hide an inappropriate post, I would be happy, and experienced users would get more say then new users.
3 Likes
I strongly second this, I meant to reply a lot sooner and chase up getting more info on the forum-specific settings for where I moderate, but unfortunately some very serious personal stuff got in the way.
It would also be great if this information was also visible to moderators please, who are going to be the ones interacting with members and flags in most cases on most forums.
Saying “three flags hide a post” was something everyone could understand, without any problems, knowing they could drop a flag to say “hey, I’m not sure about this”, which would bump it right to the top of the list for the next Staff to log-in, and that if 2 other members also felt the same, it would hide the post, but no-one’s actions alone would cause anything drastic.
Something that is equally easy to explain to members, for that so-desirable member-moderation aspect unique to Discourse, would be really nice, if possible please.
Meanwhile, a trustworthy TL3 member whose flags I intentionally Ignored to get his weighting down to 51% was still able to hide posts with a single flag, and this issue has continued to cause periodic problems on topics where passions were already running high (hence the flags) when people see their reply immediately getting hidden, which has significantly increased moderation workload.
Making those visible to, not editable by, moderators, who as I said are the main people involved with flags on a day-to-day basis, would give me a helpful response where at present I cannot provide one; I requested screenshots of our settings from my admin but he’s been busy and so have I, so your post is describing a thing I unfortunately cannot even see. 
I do not believe we have any custom settings, though.
Adding flag purgatories to users whose flags are not Agreed at a high rate will mean we have to start Agreeing flags which were just to alert mods to check something out, and risks creating a new pitfall. Especially if it’s only visible to admins (sorry to keep harping on that one).
Therefore, if this kind of thing is created, please consider giving moderators a manual way to see who is in flag purgatory, and to increase someone’s weighting back up (or down) manually? Moderators can already change and lock Trust Levels, and this is very similar.
4 Likes
Just want to say that I strongly third this. 
The way that @Ubik put it rings true for me: while the new system is smarter and might make correct decisions in the long run, it is much harder to explain to members. Many members in my community do not understand the new changes, which in turn causes lots of confusion. This confusion has caused animosity between some members. The prevailing conspiracy theory is that there is a band of malcontent members flagging new posts to hide them, when in reality it was a single flag that auto-hid the post.
As @Pekkanen said, if we could get to a point where 3-5 flags hide a post, that would be ideal!
3 Likes
Most of this discussion has, naturally, been about statistics and settings.
However, something I need to cover because it’s very important is that flags are not always a binary issue, “Agree with flag because the post needs to go” versus “This user has flagged wrongly and needs to be downgraded,” with no middle ground. I’ll give examples below:
From my own experience, users who flag the most are often pointing up minor things before they become problems, or requesting a check on someone because they spotted word use matching that of a previously-banned miscreant, or maybe they just want mods to look over a topic and see whether it’s in need of some guidance.
The members whose flags are most helpful because of this kind of sensitivity are not always going to have every flag Agreed (and as I understand it, Agreeing a flag has an effect on the flagged user’s progression to TL3, so Agreeing is not an appropriate default action).
The forum I moderate has been actively engaging our TL3 members with the moderation process ever since we moved to Discourse, we have regular conversations by PM and through topics in the Lounge about matters relating to moderation and how the rules are applied etc., Regulars know they can ask questions, query the precise reasons for any decision, and give feedback & suggestions, and this has been extremely successful and well-received.
Having flags cast for the smallest hint of suspicion of a problem has significantly reduced moderation workload, by catching things early, and as a result the forum is sure to have over a dozen people with what amount to mod-lite powers, and the confidence to use them, online 24/7/365. This has kept our forum stable and cohesive, even as our membership expanded rapidly since the move to Discourse.
It’s also highly effective: Regulars getting involved because they’re confident using flags means that we’ve seen would-be trolls and spammers cut off within a few minutes, and because they see the positive effect of being able to engage as guardians of the forum, Regulars have also stepped up in a big way to guide newer users and show them the ropes, which has had a positive effect on the entire culture of the forum.
TL3 Regulars also know they can take or leave this, it’s not a duty they have to fulfil, but they have tools to hand the moment they spot a problem.
It’s nigh-on the perfect system, and it all hinges upon flags being 1. comprehensible and easy to explain, as they used to be, and 2. members, including TL1 & TL2, being encouraged to feel confident that nothing happens as a result of a single flag, so don’t worry about errors, just act when you see anything that concerns you.
Experienced moderators have false positives pretty often (just like the TSA, doctors, and anyone whose role it is to look for problems): we suspect someone, but then that turns out to be incorrect, and members can expect to have more, not only due to inexperience but also because they can’t check IP addresses or see someone’s previous Deleted Posts, so encouraging people to be confident about using flags does require the assurance nothing bad can happen, to anyone, as a result of a single flag.
And finally, flags becoming more of a “do or die” thing, with one of the two users involved risking loss (X% closer to flag purgatory/one-fifth of a strike against TL3 for a year), presents mods with another level of information to bear in mind, and set of decisions to make, when a flag is raised:
- What do I need to do about the flagged post and/or person who made it, which is part of the job, but now also
- Do I Agree this flag against someone who didn’t break a rule, or diminish the weighting of the person who flagged in good faith?
Untangling some contentious topics or disputes can be time-consuming and complex enough anyway, without that added consideration! 
Factor in that many users of forums do not have English as their first language, and may be reading posts by other users halfway round the world for whom English is also not primary, and that adds another reason flags may be raised, with sincere intent, against a user who’s done nothing wrong.
With those complexities in mind, if the following isn’t possible (or desirable):
… please keep Ignore so that neither the user flagged, nor the user who raised the flag, see any change in weighting or progression towards a higher Trust Level. Ignore would maybe more usefully be termed “No Action,” even?
6 Likes
So this happens on the BBS as well, where the “something else” flag is often used to “bring something to the attention of the mods”, which may or may not include an intention to add flag weight to the post.
I wonder if offering the capability of the something else flag type to send a message to the mods should be decoupled from flagging a post, in the same way, the “I want to send a private message to this user” is (same popup, different effect).
6 Likes
I don’t think there’s a problem this will solve, because the issue is that the other flags are doing too much, not that Something Else needs to do even less. 
Creating tiers of alerts and then conveying this to all members (not just Regulars, and many of whom are using online translation or have limited English) adds another layer of time-consuming work for Discourse moderators, many of whom, like me, balance volunteer modding with busy lives, and also another change for users to adapt to, with no clear benefit.
For example I would have to edit a huge load of posts made where I and others have explained how flags work over a lengthy period, posts which could still be found by a new user searching for help, and even after I’ve done all that, there remains the danger of a user who’s not aware of the change doing something wrong, being embarassed when this is explained, and then deterred from future engagement with flags.
(I’ve been trying to refrain from saying “if it ain’t broke, don’t fix it” all day but I kind of have to say it here.)
Also, I do not always rush to click a new PM notification if I’m doing other things in different tabs (like paid work unconnected to the forum, or answering on a topic, relevant to the subject discussed) but will always attend at once to a red flag icon.
My actual dream scenario would be for TL3 to have an additional flag option on the menu, which IS configured to hide a post with a single click, and which sends a notification PM to Staff (and consequential email, if not logged in) saying something like “Rule violation: I have hidden this post pending Staff review” with a field to add any other info, and then each forum can decide which specific rules (beyond obvious things like threats) they want to tell their Regulars to use this for.
That would give the most frequently-needed & useful capability of Discourse TL4, without requiring careful training of users regarding what they can/should edit, and when & how to use the other functions like topic merge, split, close, etc.
And hell, while I’m at it, for mods to have the option to get a PM generated by ALL flags, containing reason (Inappropriate, Off topic etc) and the subject of the flagged topic/post + oneboxed size excerpt.
This isn’t about having the PM, obviously, it’s about the email it generates when logged out, which assists triage and time management. I can explain more about how this would help if anyone wishes to know.
3 Likes
I’ve been held up on a bunch of other tasks but if all goes well I’ll make some movement on the issue of stuff being hidden with one flag early next week.
5 Likes
So the idea is, secret collusion to flag the same post, even though flags aren’t visible? Behind the scenes there could be implicit agreement to flag in lockstep? All they’d get out of that is success in hiding the post rapidly, though, because these flags would eventually be reversed by staff, hurting flag weight for all flaggers, so they do pay a long price for this behavior in terms of reduced flag weight?
An advanced solution would be to notice the collusion and add an additional penalty but that is very complex.