Upgraded to 3.3.0.beta3-dev yesterday and also Installed AI plugin. The plugin is currently only enabled to staff member (5 people) But the entire site is dog slow, I’m getting extreme load errors. I can’t seem to figure out where is it coming from my server load seems fine. Is there a place or places I can go to figure out what is causing it.

[image] Jose C Gomez: Would spinning more unicorns help? Yes, that would probably help. env: UNICORN_WORKERS: 8 in the app.yml will do this. I recommend pulling performance numbers using the prometheus plugin if you have that set up, or you can use performance headers. Analysing your web logs should help a lot with identifying why your server is so busy; looks like crawlers are a good place to start.

Here’s what I see in the Crawler Report not sure if tis bad or good or what. I don’t have a frame of reference. [This image displays a line graph that shows the daily, weekly, and monthly web crawl pageview metrics over a period of time, along with options to view different date ranges and export or refresh the report. (Captioned by AI)]

Looking at my Server looks like unicorn processes are pretty busy [This image displays a terminal window showing system resource usage, including CPU, memory, and process information for various tasks such as PostgreSQL, nginx, and unicorn workers, which appear to be related to a web application or service. (Captioned by AI)] Is this the cause? Do I need more CPU? or Just more Unicorns?

Has its been a while since your last upgrade? Maybe it’s doing some kind of image processing or rebaking. You can have a look at /sidekiq to see what it’s doing.

Queues Are Empty [image] I don’t really know what the rest of this means. [image] [image] I’m not sure what’s normal here… Here’s our server spec [image]

Rebooted everything was back to normal but now we are getting extreme load again. I can’t figure out where the issue is coming from is there any tooling that can help within discourse?

So the 3 unicorn workers [image] Are busy… but we aren’t getting higher than normal traffic as far as I can tell it is about the same it has been the only change was upgrade to 3.3.0 and Added Ai plugin but its only available to Staff Issues Started Yesterday 6/3 [image] We do have a few more crawlers it seems. [image] Here’s just Crawlers over a month, but again ti doesn’t seem that much higher. The site is almost un-usable [image] Any help would be appreciated!

This is a guess, but the only thing that stands out for me in the Sidekiq logs is that the job that’s shown is NotifyMailingListSubscribers. That job can potentially create a lot of requests. Also, do you see any errors on your Admin / Logs / Error Logs page?

I added a block to the facebook crawler cause that guy was going to town [image] However I noticed that adding slow / crawlers isn’t updating my robots.txt [image] but robots.txt doesn’t show the slow entries, only the block entries. [image]

[image] simon: Also, do you see any errors on your Admin / Logs / Error Logs page? Quite a few of these [image] I see 3 errors but they don’t seem related… (though it is hard to tell) Job exception: PG::DatetimeFieldOverflow: ERROR: timestamp out of range: "271768-09-23 06:24:11.793040 BC" LINE 1: ...sers"."moderator" = FALSE AND (users.created_at < '271768-09... ^ ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="10.10.121.119" HTTP_X_FORWARDED_FOR="14.140.10.244, 14.140.10.244") app/controllers/topics_controller.rb:1298:in `track_visit_to_topic' app/controllers/topics_controller.rb:169:in `show' app/controllers/application_controller.rb:422:in `block in with_resolved_locale' app/controllers/application_controller.rb:422:in `with_resolved_locale' lib/middleware/omniauth_bypass_middleware.rb:64:in `call' lib/content_security_policy/middleware.rb:12:in `call' lib/middleware/anonymous_cache.rb:391:in `call' lib/middleware/csp_script_nonce_injector.rb:12:in `call' config/initializers/008-rack-cors.rb:14:in `call' config/initializers/100-quiet_logger.rb:20:in `call' config/initializers/100-silence_logger.rb:29:in `call' lib/middleware/enforce_hostname.rb:24:in `call' lib/middleware/request_tracker.rb:291:in `call' And another job exception around SMTP

[image] Jose C Gomez: robots.txt doesn’t show the slow entries, only the block entries Discourse does it’s own rate limiting, it doesn’t rely on robots.txt

Extreme Load Error After Upgrading to 3.3.0.beta3-dev Yesterday (on Prem)

Product Support Self-hosting

Jose_C_Gomez (Jose C Gomez) June 10, 2024, 1:22pm 14

Well upgraded to a new DO instance doubled the ram and CPU. Added 8 unicorns (vs 3) did a Db reindex and vaccum and I think we are back in bussiness!

Thanks for the help

Topic		Replies	Views
Due to extreme load, this is temporarily being shown to everyone... when it's not really the case Self-hosting server-resources	19	2059	July 21, 2023
After yesterday's upgrade, my site is taking 50 seconds to open (and then behaves normally) Product Support	22	768	December 18, 2024
High CPU usage (Ruby) Product Support server-resources	20	1139	February 3, 2025
Help! Discourse workers dying on our community site Self-hosting	7	1696	September 26, 2017
How serious is "site is under extreme load, search is disabled, try again later" Product Support	16	2301	August 19, 2019

Extreme Load Error After Upgrading to 3.3.0.beta3-dev Yesterday (on Prem)

Related topics