Biggest/busiest Discourse forums?


(AstonJ) #1

Just wondering what the biggest Discourse forums are… and the busiest (concurrent users or posts per day).

Anyone got any ideas? :sunglasses:


(Robin Ward) #2

It changes but right now the top two sites we’re hosting are both gaming related: Turtle Rock Studios and Gearbox Software.


(AstonJ) #3

Nice :slight_smile:

Would you be able to share some usage figures like number of unique visitors a month and the equivalent of page views a month? I’m particularly interested in what kind of server/resources Discourse instances of that size require.


(Robin Ward) #4

I’d love to share that information, but I’d need the permission of those forum owners, as those metrics are their business information and it’s not my place to make it public :smile:


(Dean Taylor) #5

FYI: There is “some” public stats/information available on almost all Discourse installations about pages at the bottom:




(AstonJ) #6

No probs Robin, I understand :smile:
Yeah I saw that thanks Dean :slight_smile:


(Alexander Azarov) #7

Hello,

So I want to pop up the original question. Are there any real big / busy Discourse installations out there?

I am considering migrating an existing big forum into Discourse, so the first question I have “whether will it fly”. The forum I’m talking about is 20M posts, 400K users.


(Ringgit Malaysia) #8

wow, that’s some forum you have there! This will be a great reference story for Discourse if you successfully migrate them over!


(Alexander Azarov) #9

I don’t know yet if it’s even possible, ha-ha. That’s why I’m wondering if someone running any big forums on Discourse. Or at least if someone did any synthetic load testing w/ lots of data.


(Matt Palmer) #10

Based on a quick sample, I see one hosted forum that has 4.4M posts and about 350k users. We don’t do anything special to host it, it’s “just another site”. I don’t know if that’s the biggest of either of those numbers, it’s just one I pulled out of a quick inspection of a few I know are “biggish”.

Really though, it isn’t about the amount of data on disk (despite that being the easy to measure bit). PostgreSQL will, in my experience, happily manage datasets orders of magnitude larger than the 35GB or so needed to store that many posts and users. The money shot is visitor numbers and activity patterns.

More precisely, it’s about peak visitor numbers and activity patterns. If you’ve got 1,000 users, 90% of whom all come and sit on your site for a few hours a year and are all hitting F5 every 10 seconds and frantically typing posts to each other constantly, that’ll require a lot more resources than a 400k site that typically gets visited by each user once a week to glance at the “new topics” list.


(Alexander Azarov) #11

Thank you for the example.

From my own experience the load comes from page views indeed and almost never from posting / liking, but that heavily depends on the code and internal optimizations. We’ve learned that hard way, when some inefficient feature rollout or MySQL execution plan sudden change results in few hours of downtime.

DB memory footprint isn’t something I’m afraid of. It takes what it takes, we currently have a good margin.


(Alexander Azarov) #12

Matt, I have an idea here that I could compare the performance in terms of hardware required. If your hosting has few such 5M forums it could be (very roughly) comparable to our load. Could you share your hardware? If this is not a secret certainly.


(Régis Hanol) #13

You’ll find plenty of information in the hardware category of our blog and especially in the (up to date) post about the discourse servers :wink:


(Matt Palmer) #14

We’re pretty good at staying on top of performance problems in the database; we certainly notice quickly enough when someone pushes out a query that’s a bit suboptimal. Also, not using MySQL protects us from a spectacular range of facepalms all by itself.

Oh, we definitely have a lot more than 20M posts and 400k user accounts summed across the sites we host, but they’re hardly comparable, because each of our Enterprise customers has their own PostgreSQL cluster, some of which are running together on the same (Really Big) machines, but often they’re on different physical hardware altogether – it just depends on the luck of the deployment draw. So you’ll have trouble doing any sort of aggregation-and-comparison.

For all that, I’ll say one thing: if you’re on our Enterprise hosting, and you have problems at scale, we will fix them, and fast. I also think you’d have trouble running a site the size of what you’re describing for $1000/month in manpower costs, let alone the hardware, network, space, and power costs involved. We can do it because we amortise all the costs over the thousands of sites we host. For a single large site, it’s going to get expensive real quick.


(Tom Newsom) #15

Well presumably they have the manpower and budget in hand already. It depends on what sort of community they have - I used to help run a 14m/50k vbulletin site that transitioned to xenforo and that’s run entirely on a volunteer/donation basis.

aazarov: What forum software are you currently doing 20m/40k with?


(Alexander Azarov) #16

Almost 20m and almost 400k to be precise, we’re pretty close to these numbers.

It’s a mix of different generations of our platform, we upgrade some parts and they live side by side with much older components. Architecturally it’s CQRS. Nginx, PHP and Scala backends, Angular, Scala middleware (workers doing heavy jobs), MySQL, Redis, memcached, some MongoDB, Sphinx search.

It’s a mess, but it works. We have resources to keep it running and very good experience, but we’re not satisfied with the velocity we’re evolving. Our community deserves more features at faster pace. That’s why I am investigating completely another idea to take e.g. Discourse as a solid foundation and enrich it with our specific plugins for our community.


(Alexander Azarov) #17

Thank you. Certainly I understand what you are selling. This can be an option for someone else. I sincerely wish you more customers.


(AstonJ) #18

What hardware are you running at present?
What is your monthly server budget?

I would put up a test Discourse forum (after importing all posts/users) on a .net or .org domain and redirect all traffic to the DC site for 24 hours to get an indication of whether it is a good fit. Obviously keep your members informed and just let them know you are trialling a new platform for them.

Are you able to share a link to the forum? I’m curious.


#19

What are currently the most busiest forums?


(Matt Palmer) #20

Have you read the previous posts in this topic?