Our old pal @hanzo clued me in to http://civilcomments.com/ which I couldn’t make heads or tails of from the website alone. But watching their intro video tells the story.
Here’s the special bit : before anyone can post a comment, they must first rate three random comments for quality and civility
and rate their own comment for quality and civility, too:
Then all that “human ranking” feeds into the other stuff like explicit likes, replies, etcetera to determine which ‘best’ comments get published
I’m kind of unclear if you have to rate 3 random comments every time you comment, but maybe? Perhaps it’s only other new users rating other new users?
It’s a Disqus competitor, not anything that really competes with us. But some interesting ideas at least. One red flag for me is rating yourself on “quality”. Because I don’t know about you guys, but all my posts are ∞ quality.
I don’t mind if the comments are uncivil, as long as they are useful (i.e. have good quality).
(This is not to say I dislike civility, I just don’t mind that others are uncivil as long as their posts are useful).
If the feature was available, I’d hope that the Civility part could be disabled.
It’s up to each community’s mods and users (who can flag comments if they’re way below that community’s standard) to define civility.
Well, that’s another problem. You have to set ground rules about what civility means. I do think there are accepted normal standards for civility, but you probably have to be somewhat explicit about teaching those standards to people before asking them to blindly rate others on it.
I always found it amusing that Joel ran an experiment on this using Mechanical Turk and rating Stack Overflow comments as “friendly”, “unfriendly”, or “unknown/neutral”
Of 7000 comments submitted, there were 161 that were rated as “unfriendly” by 75% or more of the reviewers… that’s about 2.3%
Of 7000 comments submitted, 557 were rated as “friendly” by 75% of more of the reviewers… that’s about 8%
I also think friendly/unfriendly is a lot easier to decide, and even there, consider that of 7000 comments:
557 rated friendly
161 rated unfriendly
6,282 rated neutral/unknown
Which kinda shows you how hard it is to rate things on squishy subjective emotional stuff.
Remember, too, that in the video they’re asking you to rank civility on a scale which is harder yet! Was this 0/3 civil? 1/3 civil? 2/3 civil? 3/3 civil?
Very interesting. Could be done randomly to check the current environment’s social health. Or a requirement for basic users before ascending to the member trust level.
This has got to be part of the internal mechanism used to build a credibility or trust rating. If you consistently say that your own posts are A+++, and the broad majority of the users think otherwise, the algorithm could weight your opinion less over time. What muddies that, however, is a cesspool of trolls.
What happens when you are the only shining star in a sea of shit? If you are the only one in touch with reality, and genuinely honest with yourself about the quality of your posts/comments, wouldn’t a community filled with trolls that feel the opposite simply work against this algorithm?
Of course, if self-rating is used in a different capacity, it escapes me as to how. Perhaps I’ll have more on this after .
before anyone can post a comment, they must first rate three random comments for quality and civility
I feel like this is asking too much of the Average Joe. I don’t expect newcomers to help me keep my clubhouse clean. That’s the regulars’ job, who’ll do a better job of it anyways.
Thinking about it a bit more, there are a few things you could measure here with the force user to rate their own reply before posting technique:
Did they …
change anything before posting? How much did they change?
abandon the response entirely?
spend any time looking at the dialog before clicking “post”?
I agree that the self rating is probably a placebo (or maybe even a red herring!), it’s doubtful you would factor any self rating into your algorithms because of bias. But there are a number of interesting data points you could collect here…
I’m not even sure whether the main point here is to gather data. I think this is also a way to ensure the 98% of people that usually post civil comments, but could be tempted in the heat of the argument, are thoughtful and civil: Is there a better way to teach people to self-reflect on their comment than to show them other comments in the context of rating civility, and then letting them look at their own comment again?
.