While studying basic statistics of various communities for a chapter in my book, I made a discovery: healthy and active communities may demonstrate very different metric results. So my conclusion so far is that you should not rely on basic activity stats to measure healthiness of an online community.
I calculated activity per active users in last month:
Topics created per user vary from 0.07 to 0.55 – this makes a huge difference, i.e. 7 new topics vs 55 new topics per 100 active users per month.
Messages created per user vary from 1 to 30, i.e. 0.1k vs 3k new messages per 100 active users per month.
Communities examined are those in different industries, in 4 different languages (English, Spanish, Russian, Ukrainian), and spanning from 1k to 1.5M registered users.
Also, I noticed that the percentage of registered users which get active in last month vary as well: from 0.3% to ~50%.
Sharing this in case it can be useful to someone.
P.S. Of course the Discourse team has a greater insight based on the anonymous statistics they collect from Discourse instances. Yet it looks like analysing 10-20 random communities already does say that activity metrics are nowhere near being signs of community healthiness.
Not sure; I’d be curious to see what @hawk has to say.
One thing I will note, if you have 5,000 people posting one new topic a day, no human being could possibly process that much information on a single site. I think there is a lower limit of “reasonable activity to keep people coming back” that is far, far more important than “every user must post once a day”. Quality always trumps quantity as well, so I’d rather have 5 really interesting topics versus 50 “I typed stuff in a text box on the Internet” topics.
Interesting analysis. Stupid question: you seem to imply that all the communities you looked at are “healthy and active”, but how do you know that?
Putting it differently: you are saying that the differences it the metrics of these communities may not reflect differences in healthiness or “activeness”. But in order to say that, you’d need some different measure of healthiness and activeness to compare the discourse metrics with…
Very basic: has been active for quite a long period of time, with posts spanned regularly over time (not seeded in just one period of time).
I also preferred forums from two resources:
the ones I’ve known for a few years and can say they are healthy by applying common sense
the ones interviewed by the Discourse team for their blog posts (I believe they did the job of identifying the healthy ones)
plus Discourse meta, plus my own community
And just randomly looked at top topics in last year to see how diverse they were – this is subjective, but nonetheless I think it is enough. For any type of research, you have to start with something anyway.
Fine. So if you are just distinguishing between healthy and non-healthy communities (rather than different degrees of healthiness), your message is basically that a community can be healthy with as few as 0.07 topics per user and/or 1 message per user.
Fair enough. But for some reason I also read your OP as suggesting that it doesn’t really make a difference if a community has 0.07 or 0.55 topics per user etc. But that claim would not warranted be by your data (because you are not distinguishing between different degrees of healthiness).
I’m not saying that it’s necessary to distinguish degrees of healthiness, but if you’re distinguishing healthy from non-healthy, it might be helpful to also look at some unhealthy ones. Comparing the metrics of “obviously healthy” and “obviously non-healthy” might provide some orientation as to what might be a threshold that tells healthy and non-healthy apart. And if you find that the metrics of healthy and non-healthy are not significantly different, then that would indeed be a reason to caution against misinterpreting them as indicators for healthiness.
Thanks, sounds reasonable; but this is a different story.
The original goal was to see whether “obviously healthy” communities (love the term) have these basic metrics similar or not.
To be fair, I’m not a big fan of plenty of data. The more numbers you have, the higher level of noise and the more wrong correlations you can find (e.g. see analysis of noise in data in “Antifragile” by Nassim Taleb, with plenty of links to other books). I prefer action and empirical knowledge. Just wanted to make sure I don’t miss anything with those basic numbers.
Right. We’ve been trying to educate community builders to step away from vanity metrics for a while now because they’re not necessarily indicative of health. That said, every community needs engagement of some type in order to achieve its goal/s, whatever they are. Fluctuations might also indicate an issue that needs to be addressed so they’re important to monitor at a high level.
But engagement stats aren’t comparable across all communities. You need to take into account community type. A support community for a popular product will look very different to a niche community of practice for instance. Both might be ‘healthy’ but those numbers will vary greatly.
I’m not sure there is a one size fits all metric. For a CoP I’d look for a high DAU/MAU score to indicate health, but a similarly high score for a product support community would be dire.
might be relevant… or not… but i wonder if there is any relationship between community forums, “successful” levels of engagement, and dunbar number theory? it’s not apples to apples, but, it fascinated me for a moment.
We see this as well. While I agree with hawk about not focussing on vanity metrics, it really depends on the goal you are setting for yourself.
Most importantly, in my experience, these numbers only tell you something if you take the history of them into account.
Having 100 active members doesn’t mean much.
Always having 100 active members and suddenly seeing an increase to 120… that definitely warrants an investigation
