The MEGATOPIC: public good, or public menace?

When we founded Discourse in 2013, I had the idea that we’d support topics of any length or size, from 10 replies to 100 to 10,000 to 100,000 or even more!

A few years ago we ran into a technical limitation where we send down a list of all the post IDs in the topic when you enter the topic. This starts to cause problems on the client – particularly older smartphones with less memory and CPU power – at around ~10,000 replies, so we created a site setting that automatically closes topics at 10,000 replies and defaulted it to on. It’s certainly possible to turn this setting down or off, but we believe in “safe by default” at Discourse, and extremely large topics were no longer safe, at least for some clients.

This was meant to be a temporary solution as it was a significant amount of engineering effort to create a solution where we sent down only a partial list of topic ids based on what you are viewing. We planned to revisit this later.

However, as we started to investigate MEGATOPICS, it became clear that they are

  • not that common
  • often quite problematic

… and now after looking at this for quite some time, with many live examples, I believe that MEGATOPICS should, in fact, be permanently discouraged. Here’s why:

  • Megatopics are impractical to read. By the time you reach 10,000 replies, who in their right mind is going to sit down and read through all 10,000 of those replies? So how does anyone catch up with the current state of the topic – maybe by reading the first post and the first 50 replies? Reading only the last 100 posts? Even using our handy “summarize this topic” function, you’re looking at reading 1,000 of the most liked replies to understand what went on in that topic. Your best bet is for someone to read the whole thing and edit a summary into the first post, which would require a staff member, or the first post to be a wiki.

  • Megatopics are impractical to navigate. How long did it take this topic to get to 10,000 replies? A week? A month? A few years? While you can easily jump to the beginning and end of any topic using the handy timeline feature in Discourse, you are on your own figuring out what exactly happened in the vast middle of that topic. We’ve discussed adding breakpoints or chapters to topics at some point, but there were certainly important “events” in any long running topic you’ll need to dig through the dates and posts to figure out yourself.

  • Megatopics are genuinely rare “in the wild”. Right now I’m looking at the archives of a forum we migrated to Discourse in 2017 that dates back to 2003, and sorting by replies, I count 7 topics out of 87,602 that are over 10k replies. Seven! That’s about 1 megatopic for every ten thousand topics. There’s also a fairly steep dropoff in topic length at that point; only 9 additional topics are over 8k replies.

  • Megatopics tend to lose control of the actual topic. It is normal for a topic to veer and meander, but the longer the topic, the more likely it is that gentle conversational curves morph into radical changes. Furthermore, the longer the topic goes on, the more the world changes around the original opening post that kicked things off – if it is about a then-current news event, for example, other more recent news events may have rendered it moot or significantly altered its meaning and trajectory by now.

  • Megatopics are often users expressing a desire for a chatroom. There’s nothing wrong with the desire for a chatroom, but chat works best in actual chat software, which is designed to host impermanent, rambling, neverending conversations about anything – and where users can create their own channels at will. For example, here’s a topic titled Ask me [warhammer] lore questions which currently has 58,283 replies… and counting.

So here are my recommendations:

  • Use natural breakpoints in topic titles: years, seasons, months. A topic titled “Best Horror Movies” will be hard to find and hard to navigate. But a topic titled “Best Modern Horror Movies” or “Best Horror Movies of 2018” will fare much better.

  • If something significant has changed, start a new topic, and link to the old one. Instead of adding to the “Diablo III” topic, create a new “Diablo III – Reaper of Souls” topic to cover the game’s latest expansion and patches with the changes to the core game as a result. In discussing the Houston Oilers after their 1999 move to Nashville, create a new “Tennessee Titans” topic.

  • For big timeline events, consider creating a new topic. Big events are often worthy of their own topics. A huge release like Smash Brothers certainly deserves a new “Smash Brothers on Switch” topic where people interested in that can easier find it, versus being buried deep in the general Nintendo Switch catch-all topic. This can be linked both to and from the general Switch topic (or the general Smash Brothers topic) as needed, of course, and now it can be found easier via search as a standalone topic.

  • If your users are implicitly telling you they want chat, maybe set one up? It’s perfectly fine for Discourse to live side by side with ephemeral chat, and it’s exactly how we use Discourse internally! Chat doesn’t compete with the longer form conversations on Discourse, it is highly complementary to those conversations. There are plenty of great free chat options out there, like Slack and Discord.

67 Likes

I agree - on a forum there is limited use for a 10,000+ post Topic.

I would just have it so that when it reaches 10K, it’s automatically closed and the system posts a continuation Topic and then links the two. (Perhaps with a mod flag to bypass the restriction).

5 Likes

There’s also the use case of megatopics as landfills. The admins want you to stop trashing the forum with a bunch of garbage, so they’ve given you a landfill to dump it all in.

If off-topic chatter is becoming a problem in a community, instead of relegating it to a single topic within an existing category maybe create an entire “off-topic” category or sub-category. This at the very least grants the content some ability to have structure.

19 Likes

This is on point.

I recommend doing this, locking it to trust_level_0 to hide it from search bots, disabling badges and hiding from latest.

16 Likes

I honestly see no reason as to why this shouldn’t be left in the hands of those who are in charge of the forums. If they want topics to be over 10k then they should have topics over 10k.

Especially if it is a forum where such a thing is quite common.

I guess people can just make “[topic title] part 2” and so on but at that point you gotta wonder why not have it be 1 thing.

With this change we now have a topic sitting at 61k that is locked. Ho boy would that be a lot of parts.

4 Likes

At the end of the day it is up to you, but this topic is about ensuring you understand the ramifications of the choice you are making for the members of your community.

While Discourse is just a part of your life, we live and breathe it and it would be remiss of us if we didn’t ensure that everyone is armed with the knowledge to create the best experience possible for their audience.

17 Likes

For curiosity sake where in the settings can we change the default?

4 Likes

For our hosted customers this is suppressed unless you are an enterprise customer. Allowing megatopics in our multisite cluster causes performance problems so we just restricted this. Note, we do plan to change it so you can increase the PM limit up to 10k.

9 Likes

I’ve used this in the past. In my case, I also muted that category for all members and posted an announcement explaining how you can un-mute it if you’re interested in off-topic discussion.

6 Likes

Agreed. I think the following additional things should happen when a megatopic gets automatically closed:

  1. A continuation topic is automatically created. E.g. “What did you have for lunch today?” gets automatically superseded by “What did you have for lunch today? (2)”. This would work exactly like our Linked Topic creation, along with a closing notice linking to the new linked topic.

  2. If tags are enabled, the topics would also be assigned a generic tag.

1 Like

Eh, maybe – that’s institutionalizing a) the continuation of exactly the same thing, which might not even be the right call and b) bad topic naming.

Having zero friction here is, I think, the wrong call. How about something more like a hint:

This topic was automatically closed because it was super duper long, like seriously!

Feel free to continue the discussion in a new topic.

Where “new topic” is equivalent to pushing the new topic button.

6 Likes

Chances are if a Topic reaches 10,000 posts - 10,000 is not the natural limit. By closing it you are shutting down something that members, more than likely, want to continue.

Maybe just add a mod setting in addition to what @erlend_sh has said “do not continue topic if reaches threshold”. I would also add an admin setting too (I’d probably want to keep the max to what may be more optimal for the server - so maybe half that at 5K?).

Having said that, if simply auto-closing it is much easier to do - do that, as there’s plenty of other (more useful) things that your resources could go on :smiley:

The SitePoint vBulletin forums had a few “mega threads”. But they were not what I’d call discussions. A few were “game” threads which were more post-to-post where there was no presumption that anyone would read every post from first to last. And there was a catch-all Introductions intended to consolidate multiple “Hi, I’m new here” threads. There were also some “opinion” threads which were little more than a series of “I like __ best” with a sprinkling of substance here and there.

I guess whether or not a community wants non-discussion topics in its forum is up to the community. But in my experience and in my opinion any discussion that starts to get anywhere near “mega” has become full of tangential and repititous posts that make reading from first to last such a major task that it is highly doubtful anyone would do so.

5 Likes

The limit is already a site setting (and has been for quite a while). See auto close topics post count and auto close messages post count. If you are self-hosted you are free to change these settings as much as you wish. On discourse.org hosting we’ve enforced the 10,000 post count setting globally (as Sam mentioned above), so that cannot be changed.

6 Likes

Revisiting this after a year, and after some experience with the philosophy of decentralized social networks, I have another way to frame the problem:

Discourse didn’t want to fix the performance problem because megatopics are trying to scale the software past the human limits.

10,000 replies is clearly arbitrary. But it’s also clearly way too many posts for a single topic. It’s just like the follower / following performance limits on Twitter-like platforms: if you actually need 100k followers, consider a blog instead :wink:

18 Likes

Megatopics are bad, part deux:

As someone relatively new to College Confidential and a parent trying to help a junior through the college admission process, I find these [megatopics] overwhelming. I’m overwhelmed by the generosity of people like doschicos to provide relevant information. And also overwhelmed with the length and complexity of these [topics]. I’ve found them hard to read and even harder to analyze for the purposes of moderation.

13 Likes

Megatopics are bad if you have a forum geared towards discreet tasks with finite ends. Open ended community discussions can easily hit 10k quickly while staying on one narrow subject.

I co-admin a forum with < 500 registered users that discusses news and current events and we hit the limit frequently. We are already halfway through our third dedicated topic for COVID-19, with multiple side topics as well. The second one had to be closed after only 21 days.

Having said that, even for us it’s not a huge deal to start a new one. While I’d prefer theoretically infinite topics to be available, it’s something we can live without if implementing it is a huge undertaking.

But I don’t think megatopics should be considered a rare edge case, nor should it be assumed they are always a bad thing. Some established communities even take pride in having single topics that are both huge and long lasting.

3 Likes

Megatopics are always a bad thing. Implies you need a real chat room.

These communities are dead and dying. They just don’t know it yet because they lack self awareness.

8 Likes

Same here with our COVID-19 topic. Had to close the first after 2 months, then start a 2nd - which we’ll close at the end of 1 month. We’ll be starting a new one each month as the number of posts seem to grow very quickly. To everyone’s credit, they (we) mostly stay right on topic with only a few posts veering off toward politics, but then this has been tightly woven into politics. We also have a COVID-19 topic strictly for Research being done, and one topic just for tracking the spread of the virus, # of cases/deaths, etc.
We keep our topics to a maximum of 1,000 posts. We are a small private forum, so that saves us from the dilemma others face.

1 Like

We’ve tried having chat rooms. Our users prefer your software.

6 Likes