Peer-review functionality

approval

(Eirik da Silva) #1

I would like to propose a new approach to filtering content in online debate forums; a peer-review process. This would be drawing on the same principles as utilized in scientific journals.

When a posting is submitted to a forum it would automatically be sent out to three forum members, they would be asked to judge the quality of the post by a defined set of criteria and be given opportunity to give feedback to the author. Based on the review score the post would be either accepted and posted or declined. The review would be anonymous in the sense that the identity of the reviewers is not revealed.

This peer-review function would be intended for forums with high level of participation. At forums such as those hosted by The Economist and NyTimes one can find debates running into many hundred entries. With huge variations in quality it quickly becomes very hard for the reader to follow. With peer-review you should be able to weed out lower quality content leaving you with a debate that is easy to follow, but at the same time gives everyone the opportunity to participate.

Such a peer-review process would in a sense distribute the moderator work to all forum members. One would probably however like to leave some room for moderator/editor intervention in case something that should not have been accepted has been approved.

Such a debate would run slower than a conventional debate forum since you need time for the peer-review, but I expect many would accept the trade-off for higher quality. It would for example still be much quicker than submitting letters-to-the-editor to a newspaper or magazine.

The review task would be a new task for forum members, but it would not be a very time consuming one. One could link the number of review assignments a member receives with the number of postings that a member has, so that more active members have a higher review load. Again I think this is something I think many would agree to if they perceive that it results in high quality debates. Scientists accept the task of peer-review as something that goes with the process of science.

I can think of a number of variations and refinements to such a peer-review scheme, what I have outlined would be the basic implementation.

I do realize that it implementing this requires some work, but I do think it opens up for debates with higher quality, more participation and greater readership.

Looking forward to hearing what you think about implementing this in Discourse.


(Allen - Watchman Monitoring) #2

Hi @EirikSilva

This exists in Discourse’s Approval system, look for this admin setting, approve post count and set it to 1000 or higher.

More posts on the matter are here: Topics tagged approval

We enabled this when we wanted to maintain an excruciatingly high signal to noise ratio, and it’s drastically cut down the interaction in our forum. And by that I mean, it killed the chatter, making it almost a ghost town.


#3

I like this idea and can see how this would make Discourse a natural choice for scientific journals. I can also see how it would draw in more discussion given folks would want to either back the motion, or be critical of it - similar to the Economist here:

Downside would be the natural inertia introduced into the posting process, so I’d see this functionality as being optional and set within category options.

I can see also how more mainstream magazines could use this to crowd source decisions, as in set the peer reviewers to TL3 and approve if approvals exceed X. Same goes for letters and other contributions, even standard online comments - all could be subject to formal member approval. I appreciate we have something similar here in terms of flagging, but this would be the opposite in that certain trust levels could be trusted to peer review posts. Could cut down moderators’ work and make membership more inclusive.

Thanks for posting. You’ve set my mind going again.


(Jeff Atwood) #4

Yes the economist “formal debate” formats are the only known analog for this, and are quite involved… searching for the word “economist” here can help find the other references.