Forum Community Metrics


(Codecademy ) #1

Hey all!

I need to prove that our community forums retain customers, acquire customers and reduce churn.

Our forum works as a Q&A, bug/issue trackers and support.

We have a Pro and Free product, Our Pro product comes with live help, our free does not – which is how the forum helps :slightly_smiling: I also use the forums as a FAQ.

Any ideas would be awesome!! Is there going to be any improvements or additions to the admin analytics?


(Sam Saffron) #2

I know @Bill_S does a lot in this department, maybe he can give you some tips.


(Bill Snider) #3

Hi @AlexBCodecademy,

I do a lot of the same things you are doing with your community The dashboard should give you a good enough start for what you’re asking.

Best community practices across the industry will tell you that of all those that view your community, 10% will register and post, while 1% will stay around.

On a monthly basis, my experience is that about 1-2% of the users visiting register. Since user visits are a session count of individual users, that percentage number may actually be higher. Nevertheless, for me, that translates to over 1300 registrations a month, and almost 11,000 logins a month.

The metrics that might help you get started with your ROI (the value of your community) include…

  • Page Views = Overall community Activity shown as Total API Requests.
  • User Visits = those that are landing at your community doorstep from searches and links. These are shown as Anonymous API Requests
  • New Users = those that are registering to your community to have a look around.
  • Topics = those that are engaging the community with questions and comments. These are mostly new users (for me at least).
  • Replies (Posts - Topics) = those that are responding and by definition acting as the ongoing part of the community.
  • Registered User Visits (Shown as Logged In API Requests) = those that are new and regular users. This number represents potential engagement in your community.

If you really want to delve into the potential analytics of your community, I would install the Data Explorer plugin, I don’t know if any basic reports come with it, but if you have someone who knows PostgreSQL, you can extract other reports for your community.

I hope this helps.

[edit]

Something I forgot to add is that installing the solution plugin will also give you an avenue for measuring value. When users post a solved for their question, that is a measurable value to your community. When I used to be with Lithium, they told me that having a 9% solution rate was really good. I’d make sure to have that plugin and promote solutions.

Another thing I would also add is that community promotion is critical. Get to know what SEO means and how to do it for your community. If you have a corporate site make sure the link to your community is promoted there, and the same for your support site. The more visible your community is, the more value it can gain.


(Codecademy ) #4

So we have a “forum” open to everyone and closed off bits for our “pro users”. How would I measure the success and value of that?

I am also trying to prove that community members stay in our community (lifetime value) longer and spend more money.

Right now I want to report on:

  • % engaged (I am at 16% engaged right now level 1, 2, and 3 is what I consider engaged)
  • time to first response (to see how active and happy people are)
  • accepted solutions
  • topics with no responses (to measure activity and consistency)
  • returning community % ( current community logged in / total api requests)

BUT – how do I prove this reduces churn and improves user acquisition? Not sure…

We also have help emails. Could I say that an accepted solution is an “answered question” that averted the need for customer support in that situation? I am not sure what correlations I can get away with.

I need to prove the value of the forums for free users (hopefully user acquisition, increase retention), and Pro users (reduce churn, increase retention)

Do you have a link for the plugins?

To get a % solution rate – I would take posts/accepted solutions? We get a little over 100k new users per month.

Our community is linked in our footers, and I promote on social media. Anything else you recommend? Do you have copy you can share around how you have promoted your forum in the past?

The value of our forum for Pro users is to share the projects they are working on to get feedback, and support - as well as to show off! (for now) They get access to live chat, so the forums would be for deeper community discussions around the projects we offer.

May I also ask how many moderators you have? (as a % of users as well?)

All the best,

Alexandra Bowen
Codecademy Community Manager


(Bill Snider) #5

The same way. But you’ll need the Data Explorer to pull reports for your hidden categories. Obviously, you’re creating user groups for your hidden categories, so you can run reports on group activity per its hidden category.

This will require a user report that pulls the amount of posts read, replies posted, and/or time online, for each user in the user group. Once you get the extract you can chart it over a long period of time. This will tell you how long they are active.

I have charts where I’ve tracked posting activity on a weekly basis for years.

Some thoughts…

  • 16% engaged compared to what?
  • That requires a category report including time to response from Data Explorer. I’m not sure you can correlate that metric with how happy people are, but I guess if they’re glad someone got back to them, then you could count that.
  • Accepted Solutions - in what way? Do you want Global, or per user, or per category?
  • No responses are listed in the Unread tab. But that would also come from Data Explorer anyway.
  • Current community logged in / total api requests is worthless. You’re looking for the correlation between anonymous api requests verses logged in api requests. That percentage would be helpful.

You can’t prove anything reduces churn other than continued engagement. User longevity is your only hope of marketing value. Also, you would need to compare users across categories. Catch them moving from lesson to lesson.

I looked at your community, and your board structure is immense. The only thing you can hope to do is track users across categories, both public and hidden.

Absolutely! Solutions are prerequisite for proving value. Likes help also. However, I’ve noticed on this community that people are Like happy, so you can only go so far with Likes. They really only show that people are responding. Well, I guess that’s valuable in and of itself.

In your reporting, look for how many public users end up in your pro boards. That’s the best I can think of. However, you have to realize the your public boards are your marketing tool for your pro boards. Treat them more like that. Offer deals on the public boards users can’t get other places. That will show value.

@sam can help you with that.

No. you get percentage of solution rate by accepted solutions/topic. Derive your percentage from that. I would also recommend doing this on a weekly or monthly basis, then chart your progress if you want.

You have a whole world of SEO you need to explore. You want major searchability for your community. That’s as important as links posted on social media.

I have enough, but they are paid to moderate. So, we don’t lean on some of the Discourse moderation features. However, my moderators love Discourse for those features.

But, the amount of moderators you need isn’t based on number of users. It’s based on the amount of posts needing moderation, and the amount of back-end work the moderators are expected to do. Back-end work would include support escalations, product escalations, idea escalations, responding to zero-reply posts, user management, etc.


(Codecademy ) #6

Thank you!! Wow… :slightly_smiling:
@sam can you please help me install the Data Explorer?
This will require a user report that pulls the amount of posts read, replies posted, and/or time online, for each user in the user group. Once you get the extract you can chart it over a long period of time. This will tell you how long they are active.

Can I pull a “user report” in discourse? How do you suggest I do this? Where/what are the features?
(we have almost 500,000 users atm…so I need some quick and easy)[quote=“Bill_S, post:5, topic:37962”]
16% engaged compared to what?
That requires a category report including time to response from Data Explorer. I’m not sure you can correlate that metric with how happy people are, but I guess if they’re glad someone got back to them, then you could count that.
Accepted Solutions - in what way? Do you want Global, or per user, or per category?
No responses are listed in the Unread tab. But that would also come from Data Explorer anyway.
Current community logged in / total api requests is worthless. You’re looking for the correlation between anonymous api requests verses logged in api requests. That percentage would be helpful.
[/quote]

16% engaged global --compare to total users. So really active vs. inactive. I consider level 1-3 active and level 0 inactive/not engaged.

Accepted is global. Do you not think that is a good metric? Not sure what that proves…I am hoping it will show that a user got their answer who may have “churned” and it reduces our customer support need.

I think I meant anonymous vs total api – but what would that really show?

How do you suggest I do this? Do I need to track movement across categories, or can I just track global engagement %? How would I track users across categories? I am not sure that’s the best way to show engagement… But I am not sure.

I am not sure how to track likes…and the value of it.

unfortunately, free can’t access Pro, but Pro can access Free forums… I am not sure what you mean by offer deals on the public boards? Unfortunately, I can’t provide any discounts or incentives right now. :frowning: [quote=“Bill_S, post:5, topic:37962”]
No. you get percentage of solution rate by accepted solutions/topic. Derive your percentage from that. I would also recommend doing this on a weekly or monthly basis, then chart your progress if you want.

[/quote]

I just go under the admin analytics and get the global accepted solutions. how would you suggest I do otherwise?[quote=“Bill_S, post:5, topic:37962”]
You have a whole world of SEO you need to explore. You want major searchability for your community. That’s as important as links posted on social media.
[/quote]

I will look into this more! [quote=“Bill_S, post:5, topic:37962”]
back-end work the moderators are expected to do. Back-end work would include support escalations, product escalations, idea escalations, responding to zero-reply posts, user management, etc.
[/quote]

True that! I only have 20 mods right now, all volunteers.


(Bill Snider) #7

You’re welcome.

Yes, global is important for your overall percentage. Solutions prove effectiveness - that people are being helped being in your community. Likes show that they are relating to other users. Relating is key to building a sense of community.

Well, if you get rid of non-human requests, webcrawlers, then all you have left is anonymous and logged in. Logged in requests include new registrations and continuing users, while anonymouse is well anonymous users.

This goes back to learning how to use Data Explorer. With the right reporting, you can do this.

Maybe it’s time for you guys to change that, or use the free areas to build up the value of the pro forums. Get your superusers to do testimonials. Heck, if you’re using volunteer moderators, then put them to work - unless they’re not taking any classes.

For a global analysis this fine. The global number is what I use to show overall value. Don’t short the global numbers. Nevertheless, in order to know what’s working in your community, you will need to dive down to board (category) and user levels. Get someone who knows how to do SQL reporting to help you.


(Codecademy ) #8

Hey @sam and @eviltrout am I able to get assistance from you guys on building these 2 things? I need help installing Data Explorer and building deeper SQL reporting.


(Codecademy ) #9

To get this # do you divide Logged In by New Users?

I am hoping @sam can help me with this?

So I should only report on anonymous vs. logged in? That will show me what % of community members are “coming back”/returning or engaged? How do I track new registered vs. returning? (I assume this is in that data plugin?)

yeah I am working with my mods to reduce this - it’s a pain for sure :frowning: And I’ve received negative feedback about it.


What is the difference between logged in API requests, vs. new users, vs. user visits? Which ones should I measure, why and what does that show me? How do I compare them? (see my start at trying to break it down below)

Registered User Visits

  1. Shown as Logged In API Requests
    Those that are new and regular users.
    This number represents potential engagement in your community
    90% of community will visit, 10% will register and post, while 1% will stay around

  2. New Users divided by User Visits
    % of users visiting register
    % of new users vs. registered (logged in api request)
    Registrations a month
    logins a month


p.s. how did you learn all of this? Do you have any other resources you can point me too? I don’t want to keep bugging you :wink: I really appreciate all of your help. You’ve been a lifesaver!!

Do you have any moderator best practices? I may be trying to hire at least one…

Any other best practices for other kinds of “community”? i.e. social media?


(Robin Ward) #10

I’ve installed the plugin for you – you can access it via the admin interface or /admin/plugins/explorer


(Codecademy ) #11

@Bill_S would you be willing to take a screen shot of what Data explorer looks for you and what queries you have set up? Then I can get help with the SQL… or feel free to share some of the SQL?


(Bill Snider) #12

Sorry for the late response. I was off yesterday.

No, you would divide New Users by Anonymous API Requests. That would be your more accurate percentage.

I don’t usually report on those percentages. However, you can if you think it will help. I focus more on how many logged in, registered, and the overall user visits (including anonymous api requests).

Also, keep in mind that even though logged in represents users returning, it’s not a 1 for 1 number. One user could be logging in multiple times over a time period. So, keeping it simple with overall logins keeps accuracy questions at bay. What I have always liked to do is graph my weekly numbers (for example) over a year to see how the trend shapes up. That will tell you whether you’re losing ground, holding steady, or increasing in your community activity. You’ll experience all three of those at various times.

Years of research, asking questions, and figuring it out for myself. You would be surprised what you can find on the internet. The truth is out there…

Unfortunately, I can’t give you mine because it’s integrated with our corporate best practices, as yours will be. But this is what I got when I googled “community moderator best practices.”

https://www.google.com/search?q=Moderator+best+practices&rlz=1C1RNPN_enUS545US545&oq=Moderator+best+practices&aqs=chrome..69i57.5520j0j7&sourceid=chrome&es_sm=0&ie=UTF-8#q=community+moderator+best+practices


(Bill Snider) #13

By the way, I did find this. It might help you for moderating. Just adapt it to how you want to run your community. I had to save it as a text file though, sorry.

Moderator Policies [sample].txt (10.5 KB)


(Codecademy ) #14

it’s been so hard to find specific examples and comparable examples. A lot of people talk generally about it, or, it’s such a different use case/situation I have a hard time. :confused:

I already have pretty full fleshed out guidelines and best practices, but I always like to see how successful communities are run to compare :wink:

That’s why it was great being able to pick your brain! Thanks! :slight_smile:


(Codecademy ) #15

So we have NO internal resources for this…how easy is this to do? Could I potentially take a class and do it on my own? Do you have any resources?


(Bill Snider) #16

There are a couple of ways companies approach this: one, hiring outside firms to improve their searchability on the internet, or two, hire qualified help to do this internally. Most companies don’t think about this until it slaps them in the face. So, usually, either they pump out the money to get it done, or they assign it to someone internally to get it done.

There are plenty of courses out there on SEO, from Lynda.com to Udemy.com. Your company may even have classes on it. But, I started with Google’s best practices white sheet. That’s a good place to get started. The link is below.