How serious is "site is under extreme load, search is disabled, try again later"

AI-generated summary

Users are discussing the “site is under extreme load, search is disabled, try again later” warning on their Discourse forums. tobiaseigen is concerned about the severity of the message, especially since it appears briefly and resolves quickly. Mittineague and Falco explain that the warning occurs when server requests exceed processing capacity, which can be caused by web crawlers or high traffic. codinghorror advises checking the web crawler report and potentially blacklisting problematic user-agents. bartv suggests reviewing access logs to identify potential hacking or API abuse. tobiaseigen considers whether plugins like the Who’s Online plugin might contribute to load. Stephen notes that degraded performance can feel broken, not just slow. codinghorror emphasizes that ignoring the warnings is not advisable. thefreekick reports frequent occurrences during high-traffic events like live sports discussions. sam identifies N+1 and N+2 performance issues in the codebase and mentions that admins can adjust server resources or thresholds. TheBestPessimist asks for clarification on the term “N+1,” and michaeld provides a link to an explanation.

I’ve been getting reports of this warning message from my members. Doesn’t come up much but it is being noticed. Can I assume there is nothing to do here and it’s not as serious as it appears? “Extreme load” sounds like a huge problem. But it does go away almost immediately and search works again.

I’d appreciate any guidance on troubleshooting steps. Thanks! :seedling:

site is under extreme load, search is disabled, try again later

I’m not so sure it’s something that needs troubleshooting. But maybe it could be handled differently?

https://review.discourse.org/t/feature-when-under-extreme-load-disable-search/4538/3

You should be concerned, I would look at your web crawler report in the admin dashboard.

That will be triggered when your web requests are coming faster than your server can handle.

For example, if you have 4 unicorns and, on average, need 200ms to fully process a request, your capacity is 20 requests per second.

If, for a moment, you get 40 requests per second, the extra requests will sit in a queue waiting, and that message will be show. Also, you can’t go blindly setting up more unicorns, as the process to fully answer a web request involves a lot of moving parts that should be carefully evaluated.

Discourse gained a lot of those graceful degradation features recently as we work to scale out and handle some crazy level of traffic for the largest Discourse instances on the internet.

Looks like we did get an increase in web crawler page views in recent days. But I’m not sure what if anything there is I can or should do about it. Personally, I am ok with a notice to members from time to time when the server is under higher than normal load, preventing searching temporarily. But maybe the “extreme load” messaging could be moderated a bit, to say “server is busy”.

Blacklist the rogue crawler user-agent in site settings. There are a lot of really bad crawlers on the internet these days.

Hmm, only 1800 pageviews in 24 hours by a crawler should not put that much extra load on your server though. It might be worth opening your access log and just watch the requests come in when this happens, it could point to the real issue. It could be someone trying to hack you, or hammering your API with requests.

I Just got a report of another similar looking notification. We are doing an AMA and have a few more people than usual reading and posting in the forum, but I am not convinced that this is extreme load. Maybe a plugin like Discourse Who's Online is causing additional load?

Thanks! I will look into this now.

Is there a way to disable these in an admin setting and just let people wait if it is getting slow? I think our site is still not especially busy.

If performance degrades to the point of timeouts it won’t feel slow, it will just feel broken.

Ignoring these warnings is definitely not a good idea!

I’m getting this extreme load notification a lot recently.

I have a sports forum and it’s usually when there are people commenting on the one topic about a particular match at the same time. Is there any guide or help on what I can check to look at logs or the server to see if it’s up to speed. It wasn’t a problem until the last few weeks.

Do you think we could have regressed on logged-in user posting performance @sam?

I am seeing an N+1 on meta for sure that we need to fix ASAP

Will log some items to get this sorted. That said, once site is overloaded admins have the ability to tweak 2 things.

  1. You can give the instance more resources, (more memory for PG or more unicorn workers)

  2. You can adjust the anon threshold per:

DISCOURSE_FORCE_ANONYMOUS_MIN_QUEUE_SECONDS and DISCOURSE_FORCE_ANONYMOUS_MIN_PER_10_SECONDS

I did fix one N+1 in the discourse-voting plugin here:

https://github.com/discourse/discourse-voting/commit/9b4f233b9d33a946f626597aaf3863d2305b8809

Will see if anything else yells out

Found another N+2 here:

https://github.com/discourse/discourse/commit/accbbded15974846523c3aeca85a59454e3f2f05

Looking to see if anything more pops up.

Q: What is an “N+1” ? Although i’m a dev (java), i don’t think i ever came upon this term before.

Here is a (more or less Java-related) article that explains this issue: