I haven’t tried out the Discourse feature yet, but I do have experience with sentiment analysis generally. For instance, from a post I wrote about comments on Interpersonal Skills—Stack Exchange:
There are some fairly negative words that are upvoted: awful, arrogant, unsafe, creepy and trash. Now those almost certainly describe the content of the post they are attached to. Plenty of people have creepy ideas when it comes to human relationships. Unsurprisingly, this means that higher scored comments have lower sentiment scores on Interpersonal Skills:
score n sentiment 0 39532 -0.085 1 13925 -0.081 2 7077 -0.133 3 4171 -0.152 4+ 11278 -0.182
I also noted that moderators were more likely to delete comments with negative sentiment scores. This isn’t because moderators had access to the analysis but just because humans notice patterns intuitively. Sentiment analysis in this case was helpful for understanding some of the dynamics of the community and its moderation, but I didn’t see much reason to build it into the software.[1]
Similarly, I built an engagement model that included comment sentiment to predict whether a user would ask a second question on Stack Overflow. In that model comment sentiment was not significant. Of course that’s one model and not even for a Discourse community. But it is a data point that suggests people aren’t as repelled by negativity (as defined by the sentiment analysis system I picked) as you might assume.
There are a lot of reasons for optimism about this sort of artificial intelligence, but I’d caution against letting enthusiasm overtake us. Misapplied sentiment analysis can potentially lead decision makers into fundamental mistakes. Remember Chesterton’s fence!
Not to say doing so is invalid! Rather the advantages seem to be less direct than one would imagine. ↩︎