Experimental ‘lazy_loaded_categories’ feature enabled on Meta

[nbianca (Bianca)]

We are introducing an experimental feature to help communities with a large number of categories.

For the past few months we have been working hard on making Discourse sites work better for communities with many categories (and not only!) This will allow administrators to create thousands of categories while keeping the friendly UX you are familiar with.

This is still an experimental feature and can be enabled by tweaking the lazy_loaded_categories_groups site setting. It is currently disabled by default.

How does this work?

At the moment, all categories data is loaded when you navigate to a Discourse community. This is called “eager loading” and is not ideal for sites with many categories because it means a lot of information is transferred before the first render which increases the overall loading time.

The new method is called “lazy loading” and will defer loading information about a category until it is really necessary (for example, when searching for a category, navigating or linking one, reading a topic or post mentioning one, etc). Less data transferred means faster loading time and happier users!

What does this mean for you, as a member of Meta?

We have been heavily testing this feature internally, but given the extent of the work done, there is a slight chance of introducing bugs. If you notice anything malfunctioning related to categories, let us know as a reply to this topic or open a Bug topic.

What does this mean for you or your community?

Whether you are hosted by us or if you are self-hosted, this feature is disabled by default and there should be no visible changes.

However, if you are looking at extending your community to have many categories, we strongly encourage you to enable this feature.

In the future, we are going to enable it by default and eventually deprecate and remove the code that “eager loads” categories.

What does this mean for plugin and theme developers?

In general, there should be little to no changes necessary for most plugins, as long as they use the category components from core or preloaded data (for example, query only for categories that are already visible when loading the page, because that means they have been preloaded already).

Some problems may appear when trying to use information for categories that have not been loaded yet. Category.findByIds method will continue to return only categories that have been loaded already, while Category.asyncFindByIds may perform a lookup on the server if the information has not been loaded yet.

[Moin]

The category information seems to be missing when you use one of the links below a post to navigate to a topic in a different category. I opened Contribute a translation to Discourse in a new tab and navigated to Switching from Transifex to Crowdin, then it looks like this:

After a reload, the category appears.

[Moin]

After I posted my theme feedback topic, the category badge was missing for topics of that category

[Moin]

The search function within the composer is unable to determine if there is a matching category.

[Moin]

Is that also related to the lazy loaded categories?

[Lilly]

Yes. I have disabled the lazy_load_categories for the time being because it was breaking the docs plugin.

[nbianca (Bianca)]

We have re-enabled this experiment here. What has been fixed since it was last enabled:

• Categories in the header are populated correctly, especially when navigating from a topic to another using post links or suggested topics

• Badges styling will show correctly after creating a new topic in a category that was not navigated to before

• Docs plugin preloads categories data, so /docs page is now rendered correctly

• Other bugs that have been discovered through internal review

The only outstanding bug is about the missing search results for certain queries (short queries or queries that contain only stop words).

We wanted to make search more powerful and have extended the full-text search engine to categories, and this is an unwanted side effect. I think we will have to revert the previous search algorithm, but we’re still investigating this.

[Moin]

The categories of category moderators at /about are missing

And when I look at my drafts the categories are missing too.

The activity page of a group or user and the notifications in my profile like https://meta.discourse.org/my/notifications/responses do not load categories

When you open the composer to start a new topic with a link in a message or by clicking new message in your inbox and using the envelope at the top left of the composer to change the draft into a new topic draft, there is no option to choose a category.

[Moin]

When you open /categories, not all subcategories of documentation are shown (theme developers disappears after reload until I click documentation).

And autocomplete for searching categories does not work. Most of the categories do not show after reload

[Moin]

The categories are also missing at the list of related topics.

And when I visited the topics to load the categories for the second screenshot, I noticed that at the second blog topic, the category below the topic title was missing. I assume this isn’t important because it shouldn’t happen when the related topics work.

[RGJ (Richard - Communiteq)]

@nbianca

In general, there should be little to no changes necessary for most plugins, as long as they use the category components from core or preloaded data

This does seem to be causing an issue in one of my plugins on 3.2.1 stable.

When I inspect Site.current().categories and lazy loading is enabled, I do see (only) the categories I need (like the category for the current topic), but all preloaded_category_custom_fields seem to be missing.

They do appear when I turn off lazy loading (together with all the other categories).

So the correct category is being preloaded but it misses its preloaded fields.

[nbianca (Bianca)]

@Moin

The categories of category moderators at /about are missing

This has been fixed.

@Moin

And when I look at my drafts the categories are missing too.

Fixed too.

@Moin

The activity page of a group or user and the notifications in my profile like Discourse Meta - The Official Support Forum for Discourse do not load categories

Fixed as well.

The other problems are being currently worked on. Thank you so much for your feedback, @Moin.

@RGJ

When I inspect Site.current().categories and lazy loading is enabled, I do see (only) the categories I need (like the category for the current topic), but all preloaded_category_custom_fields seem to be missing.

It depends how the categories were loaded. I agree that there are probably some code paths that may lead to partial category data being loaded. I will communicate with the team about this issue, but since we are moving towards a state where the frontend will have to request the data they need before accessing it.

Is it a public plugin? If yes, I can have a look at it. In general, the most popular plugins we have required very little changes to make them work when “lazy load categories” is enabled.

[Moin]

@nbianca

@Moin

The categories of category moderators at /about are missing

This has been fixed.

The color of the parent category is still missing

[RGJ (Richard - Communiteq)]

@nbianca

It depends how the categories were loaded. I agree that there are probably some code paths that may lead to partial category data being loaded.

It’s happening in a regular topic view.

The only thing that might be special is that the code is in an initializer, in a widget that is attached to the post-contents:before widget. The plugin is not public but I can give you access if you PM me your Github username. Alternatively I can send you a PM with a tar.gz, which might be easier. The plugin is old and not originally written by me but I don’t see anything that’s obviously wrong either.

@nbianca

since we are moving towards a state where the frontend will have to request the data they need before accessing it.

So how will that work? I can understand the binary situation where a category is there or where it is not there. But if I encounter a category without a specific custom field, how do I know if the custom field is absent or if it is just not fully loaded yet?

@nbianca

partial category data being loaded

i.e. this should never be happening!

[RGJ (Richard - Communiteq)]

@nbianca

Is it a public plugin? If yes, I can have a look at it. In general, the most popular plugins we have required very little changes to make them work when “lazy load categories” is enabled.

So can you please help me out with this one @nbianca ?

When lazy loading is enabled, the custom fields are missing from the category object here in my public private replies plugin.

      if ((siteSettings.private_replies_on_selected_categories_only == false) || (topic?.category?.custom_fields?.private_replies_enabled)) {
        return this.currentUser && ((this.currentUser.id == topic_owner_id) || this.currentUser.staff);
      }

and I have no idea how to get them.

I’m adding the field to the preloaded category custom fields.

Site.preloaded_category_custom_fields << 'private_replies_enabled'

[nbianca (Bianca)]

@RGJ

When lazy loading is enabled, the custom fields are missing from the category object here in my public private replies plugin.

I am discussing this with the team internally, but I think we will have to revise how we preload categories for topics.

[Moin]

I found another spot where categories are missing: the AI search results.

[Moin]

When I refresh meta.discourse.org and select + New Topic, the topic template fails to appear. Initially, as demonstrated in the video, everything functions correctly. However, after reloading, the template is absent. Once I open the composer again, everything resumes working as expected.

Note: If you have support in your sidebar, you must remove it prior to reloading.

[JimmyHoke]

Based and scalability-pilled.