Defining trust levels


(finid) #11

Ok, from this, we can assign:

  1. Guest or Visitor (trust level 0) - passer-by. Not a registered user, that is, may be registered, but has not validated email. All others have registered and validated their registration.

  2. Noob (trust level 1)

  3. User (trust level 2)

  4. Junior Moderator (trust level 3)

  5. Senior Moderator (trust level 4)

  6. Admin (trust level 5)

That’s the easy part. The tricky part is assigning privileges to each trust levels. Who can post images, videos, upload text files or attachments. What can a J. Mod do. what can a S. Mod do, etc. At what level can we let a user to gain starting PM privies?


(F. Randall Farmer) #12

I’d go further. Defer setting the level numbers until after we’ve determined reasonable buckets of functionality to assign to each. Do the “easy” part last.

Don’t ask “what should a Jr. Mod do?” instead ask: What are the things we need people to do to facilitate high quality discussions?

We have a partial list already between the items listed in

and lists like [quote=“stienman, post:7, topic:3025”]

  • Able to move posts from topic to topic, or create new topics to put offtopic ones into.
  • Able to edit thread titles and categories
  • Able to close topics
  • Interact with plugins in a “gamemaster” role
  • Access to basic site stats so they can work on community building[/quote]

What else (please include from other forums, not just existing Discourse features…)


(F. Randall Farmer) #13

Agreed. All reputation is in context, and you’re right that there are different contexts when talking about content quality. I like to talk about as “The Good, the Bad, and the Ugly” - Up with the Good, Down with the Bad, and Out with the Ugly - and the trust/reputation score you use for flagging (Out with the Ugly) shouldn’t be confounded with good-citizen trust (Up with the Good.) {See more in the thread: So What Exactly Happens when you “Flag”? }

I like to think of myself in this category on most of the forums I’m in. It took me a long time to get trust on rpg.stackexchange.com - though I never thought it unfair or problematic.

Invisible - absolutely. Non-static is problematic as it becomes difficult for moderators to understand what’s happening if there aren’t clear thresholds. Let me suggest an alternative - that trust isn’t simply an ever-incresing accumulator - it decays over time and must be “kept up” in order to retain the rank. Disappear for 2 years (and your content is no longer relevant) and your trust rank falls.

From Building Web Reputaiton Systems:

…time leaches value from reputation, but there’s also the simple problem of ratings becoming stale over time as their target reputable entities change or become unfashionable. Businesses change ownership, technology becomes obsolete, cultural mores shift.

The key insight to dealing with this problem is to remember the expression “What did you do for me this week?” When you’re considering how your reputation system will display reputation and use it indirectly to modify the experience of users, remember to account for time value. A common method for compensating for time in reputation values is to apply a decay function: subtract value from the older reputations as time goes on, at a rate that is appropriate to the context. For example, digital camera ratings for resolution should probably lose half their weight every year, whereas restaurant reviews should only lose 10% of their value in the same interval.

This would need to be implemented as an override - not a bonus to trust. Note that, if full mod privs are granted by trust, this override feature is required to bootstrap a server.

Could you unpack this further? I’m not following and I like what you’ve presented so far…


(Chris Hanel) #14

Well, while not a 100% precise analogy, I look at it like a two-dimensional political spectrum.

Instead of the different axes being Social Issues and Economic issues, I look at a good forum system as having the X and Y representing the two main aspects of contributing to a forum community: Contribution through content, and contribution through administration.

On this grid, you can plot out roles similar to the four squares present in the political grid, only much more nuanced. Everyone starts off at 0,0 in the bottom left corner, and the admin sits at 1,1 in the top right. Everything in between is determined by your rating in the two axes, and as they rise and lower, it moves your placement on the grid from role to role like thus…

image

WARNING: NOT AN EXACT CHART, MADE HASTILY FOR MAKING A POINT ON THE INTERNET, A PASTIME THAT RANKS SLIGHTLY ABOVE POLITICAL PUNDITRY

With a system like this, someone who has no interest playing babysitter to their fellow members still feels the reward of contributing. Vice Versa, a person who wants to contribute through assisting with managing the community, but isn’t as experienced, can contribute through flagging and basic management and see the fruits of their labor. Some actions contribute to the X axis, some contribute to the Y axis, some to both (but not necessarily equally).

To further expand while I’m thinking about it:

  • If you want to get away from static numbers, then have the option to grade on the curve above a certain level of membership size, with built-in tenure. The top 2% of members on the scale presented here are given moderation, and it’s much harder to fall out of moderation than get in (to prevent a constantly wavering trust level for a member right on the border)
  • Should an admin desire, have a setting that informs the user and the admin that someone has earned a certain level of trust, but that promotion has to be approved by the admin first. Give the admin a chance to discuss the added responsibility with the user, or (for some communities) test them in whatever way might be proper, like a black belt exam.
  • Even more bold: Have trust levels, have roles, do everything here, but don’t have sets of features statically tied to each role. Instead, allow people to earn further and further singular features as they progress or sustain a high trust level over an amount of time. Continually reward participation by not handing over all the features to a given role right up front, especially for someone becoming a full moderator. That system would be more complex on the back-end, but my brain starts to race at the opportunity here. (and yes, I’m aware that this starts to drift towards the realm of ‘gamification’, but, I refuse to use that term out of principle.)
  • Allow people to earn an ability and title, but give them the option to turn down any added responsibility, in case it’s not their thing.
  • I could write pages and pages more on this, and probably will over the next couple days. It’s something I’ve spent far too much time thinking about.

User trust should be category specific
A new trust level: The Helpful member?
User trust should be category specific
Feature suggestion: Content quality levels/curated tab
A new trust level: The Helpful member?
(smarsh2008) #15

I’m intrigued by this discussion (followed here from Randy’s twitter feed) - if I may interject?

Is there a movememnt to the left possible? How does that work? In real relationships trust ebbs and flows, it doesn’t just ‘grow’ - but there may well be plateaus that make sense.

Also, how do you see real relationships between people in such a system (I am sorry, I think people keep forgetting about the contextuality of trust). That is, there is no single ‘trust level’ but a level of trust between people, not between people and system per se.

A little busy with a few things right now, hence a short interjection, but if you really wanted to build a system that was representative of ‘the world’ of Chris, you need to escape from the singular view of people that online systems have, and build it in terms of the multiple views of people that people have.


(Adam Davis) #16

That’s certainly an interesting perspective, but do we really want to build a social network with a complex relationship graph that knows where I stand on your specific graph, and where you stand on mine?

I think that would be too much granularity. In this system we really only need a very rough idea of the relationship between the community as a whole and a specific user. That user may be more or less trusted by individual members in the community, but the site should only need to know what the community thinks of this person on average, in order to give them more or less power and ability on the site.


(Chris Hanel) #17

I actually think that there’s a compromise to be found in these ideas that is useful without getting too insane.

Overall, I think your final level of reputation should be from the point of view of the community as a singular whole, but when it comes to a user browsing the forum, there could definitely be personal adjustments based on your own actions, such as a thread being more visible to you because the participants are those who you’ve liked and responded to on more occasions.

However, to be my own devil’s advocate, I generally disagree with going too far down that rabbit hole, as content systems that attempt to tailor and prioritize content at that deep of a level start to foster the problem of users not being exposed to ideas that are different or challenging to their own, and personal echo chambers start to form, which is more damaging to the community than anything else. (See this TED Talk for a great analysis of the problem.)


(Adam Davis) #18

ONEBOX’D!!!


(F. Randall Farmer) #19

Lots to talk about here… :smile:

Absolutely. All reputation is contextual. What we’re poking at here is what are the reasonable common contexts for forums.

Please, interject more @smarsh2008!

If we need to go all graphy, we should. But I agree, I’m not at all convinced that we’ve identified a need for that just yet. Let’s tease apart some cross-forum or cross-category reputations first.

But @ChrisHanel raised some detailed points (at my request) and we should look at each one.

Though I’m not sure I agree with the graph showing these two values as the X and Y axis, I think there is much good to mine from this basic definition.

There are at least two good wide-ranging karma scores we could calculate. I’ll name them formally to expedite discussion:

  1. AuthorContentQuality [ACQ] which derived from ItemContentQuality [ICQ] which derived from Evaluation Statements, such as Like, Favorite, Flag, Bookmark, Reply, etc.
  2. AuthorEvaluationQuality [AEQ] which is derived from either/both of Moderator Verification Statements or Peer Agreement

In fact, these are very similar to the core scores in many social media content quality reputation systems I’ve helped design/build. You can read in great deal about one here: Case Study: Yahoo! Answers Community Content Moderation [Building Web Reputation Systems]

One could go further and suggest that there exists a function (f) where:

f (AuthorContentQuality, AuthorEvaluationQuality) = ContributorTrust (CT) <-This creates a 2d graph…

But I’m not sure that we should use the combo score (CT) to assign all features/rolls. If we have multiple contexts, why not break up the roles/features along those lines? If you have a high AEQ we consider giving you more moderator tools. If you have a high ACQ - more content creation/publication tools. What would be the purpose of assigning role-descriptive text-names to each (ACQ,AEQ) sector - it just leads to confusion: “Why can’t Scholars do foo? Why can’t Jr. Moderators do bar?”

And - what are the other contexts that are reasonable for the various uses of Discourse? There are certainly arguments to be made for per-category trust scores - especially if a forum has very differing “rules” per category (SomethingAwful and Penny Arcade come to mind.) It seems that someday having per-category permissions will make sense. [I’m not rushing to add those features just yet, just wanting to make sure that we don’t design ourselves into a box.]


On to Chris’ bullets - they all deserve responses:

Static levels don’t have to scale linearly. Also, I’ve already mentioned decay elsewhere: [quote=“frandallfarmer, post:13, topic:3025”]
trust isn’t simply an ever-increasing accumulator - it decays over time and must be “kept up” in order to retain the rank.
[/quote]

Also, putting in a latch to prevent a users capabilities from wavering back in forth is technically trivial. Note that this is an edge case that doesn’t happen often, and the user should be informed whenever (s)he’s demoted and what to do about it. In short, if you care it won’t happen. If you don’t care, well - it doesn’t matter. :wink:

What about this instead? The cases are 1) Features are granted automatically (default), 2) Features are granted by Moderator fiat only. In both cases, the forum operator may opt-in to be notified whenever a user crosses a major threshold. That covers all the cases I can think of (preemptive, reactive, locked-down, and laissez-faire.)

Why have role-titles if they don’t do anything or aren’t clear? As to fine-grained feature grants on a per-user basis - I’ve seen up-close that tried at places like Answers.com. It was an administrative (and interface) nightmare - especially when you have administrator per-feature per-user override.

Again - confusion around titles - either they mean something (to everyone) or they don’t. If I see you are a “Moderator” - believe me, I have some expectations that you will act, and quickly, when you see a goatse post - abdication of authority/responsibility is not an option. Not acting in that case should lower your AEQ significantly and cause you to lose the power/title. Not to mention the possible side-effects should some content dispute erupting in the real-life “legal” arena from inaction.

Thanks for doing so! I’m sure others have similar thoughts and I know I’ve had this conversation with other companies when they are setting up their reputation systems. Honestly, you are already quite a few steps ahead of many of them in thinking…


A new trust level: The Helpful member?
(Bryce Glass) #20

From the top of my head (and, per @frandallfarmer’s direction, not limited to existing features… Privileges related to…

My own identity

  • ability to change my username (or change it more than 1x w/in a time
    period)
  • ability to change my avatar / icon (this, in combo w/the above, are identity-spoofing attack vectors)

Affect the data/metadata of a conversation

  • ability to ‘tag’ a conversation (if different than aforementioned categorization)
  • ability to add photos / videos / media to a post
  • from ‘trusted’ sources (publishers that the community has whitelisted)
  • vs. ‘non-trusted’ sources (free-form, anything goes)
  • Ability to mark a thread as duplicate / related to some other thread

Impact someone else’s karma, for good or for bad

  • ability to invite others to a conversation / vouch for invitees
  • ability to mod someone’s response up / down (‘Was this helpful, yes or no?’)
  • ability to report abuse
  • on a post
  • on a user (identity-spoofing, repeated abuses, etc.)
  • ability to ‘ignore’ (killfile) other users?

Access to deeper statistics


(Burzum) #21

How exactly is a score for a post and topic calculated right now? What’s the formula? And how is this then used to create a trust score for a user?


(Jeff Atwood) #22

Scores for posts are only used for the “Best of” topic filtering mode. They do not affect user trust levels.


(Sam Saffron) #23

Posted this tiny topic on visual styling.

http://meta.discourse.org/t/should-we-adorn-user-names-avatars-to-reflect-the-trust-level/4928


(Burzum) #24

Well, how exactly is trust calculated then? If I understood everything right there are different kinds of trust? So what kinds of trust are there and how are they getting calculated please? :slight_smile:


(Brentley Jones) #25

In that topic he covers how they are currently calculated.


(Burzum) #26

It explains it in an abstract way but I would like to know how it works from the technical side.

There is a trust_level field in the users table and a flag_level field. Based on the sample data I see that trust level is here only 1 or 2. Your link describes basically 5 of them (Passive → Active → Motivator → Caretaker → Geek God) but I can’t see where this is reflected in the code, nor how you actually calculate the score to know what level a user has gained. I don’t see any score fields by the way.


(Dan Dascalescu) #28

Just wanted to mention that a mod can manually bump a user’s trust level.

This is very useful when migrating from other forums, since many users will be established in the old forum.


(Martin Pool) #29

The definition says that you earn “Member” by “visiting at least 15 days, not sequentially”.

Does this mean, “on 15 days, not necessarily consecutive”, or that the days must not be consecutive?


(Mittineague) #30

They can be, but they don’t need to be.

For example, if a member visits at least once a day every day, it would take 15 days to get 15 visiteds.
If a member visited every other day it would take 30 days to get 15 visiteds
once a week, 15 weeks. etc.

That is, if a member “misses a day” it does not reset back to zero.

Also note that other criteria must also be met. Hence the “at least”


(Jeff Atwood) #31