Unlisted categories

Feature
1

By now we’ve dealt with a couple different customers who for various reasons would like to have categories in the thousands. The main scalability problem with regards to categories is the /categories view, e.g. this one:

https://meta.discourse.org/categories

With more than a few hundred categories, this page becomes considerably slower.

To work around this, I’m proposing a new feature called “unlisted categories”:

  • An unlisted category would not appear in any of Discourse’s category menus, i.e.

  • “all categories” drop-down

  • /categories index

  • right-hand “hamburger”.

  • The category still shows up in topic lists and users can click on the category and see all topics within it.

  • Unless the unlisted category has also opted to “Unlist new topics by default”, topics posted to an unlisted category are treated as normal will still show up normally on the front page.

  • There should be a site setting to optionally hide the “New Topic” button when you’ve drilled down into a specific category which happens to be unlisted.
    (or maybe it could be replaced with a “:grey_question:New topic” button that when clicked will explain why you can’t post directly via the forum for this particular category.

So the point is that an “unlisted category” won’t clutter up any menus, but other than that it works the same as other categories.

Example: WordPress plugin directory

As an example use case, let’s say we were building rebuilding the wordpress.org plugin directory with Discourse.

We’d want each plugin to have a dedicated category on the forum so that popular plugins could be discussed in multiple topics and the plugin author(s) could help moderate these discussions. With thousands of plugins, you don’t want to expose these categories in any sort of one-page list.

Furthermore, a formal plugin listing would be the primary landing page for a plugin, as opposed to the forum. So when a user arrives on such a page and wants to start a new topic regarding that plugin, we already know which category the user wants to post in, so that doesn’t even have to be a decision for the user. All we need to do is let the user post directly to the appropriate category via an API integration or even just a pre-filled topic URL.

Perhaps the biggest question now becomes: If our soft limit on listed categories is “no more than a few hundred, please and thank you”, what is the limit for unlisted categories? A few thousand? Tens of thousands? @eviltrout

p.s. In some cases it might be desirable to still make the category available via category search-only, as is in the New Topic category drop-down as well as the #somecategory autocompletion. I’d leave this out of v1 though.

3 Likes
2

Technically, we make some assumptions right now about categories that would have to be rethought. For example, we send the entire list of categories down with the Site object, so that could really increase the size of the initial payload if we had that many categories. We’d probably be better served to remove that assumption of categories always being present in memory, but that’s a fair amount of work.

On the back end it’s probably fine although I wouldn’t be surprised if some queries were inefficient with that many. We’d have to bump it up and see.

Having said that, what is the advantage of an unlisted category over a tag? For the plugin example you could use one tag per plugin couldn’t you?

1 Like
3

Yep, but you’d lose out on the moderator and create/reply/see privileges that you can set per category, which a requirement for at least one of our customers.

In other words, another way to solve this could be to incorporate those things into tags.

1 Like
4

The create/reply/see privileges are the key point for us here. We currently have about 100+ sub forums and cannot use a tag as the groups are private to a set number of individuals in each one.

1 Like
5

I actually disagree with this, our security model is built on top of categories, lots of complications arise when there are huge numbers (stuff probably collapses at around 10k categories at 1k categories there is a noticable impact)

This goes beyond just the category page, all the security checks, default user tracking checks and so on will join against this table.

I really dislike the idea of adding a construct of “unlisted” category especially if this is a performance optimisation.

For the wordpress example, why not simply use tags?

Also in the past I struggled with a concept of “topic moderator” but that is way way out.

2 Likes