Extreme Load Error After Upgrading to 3.3.0.beta3-dev Yesterday (on Prem)

Product Support Self-hosting

AI-generated summary

After upgrading to Discourse 3.3.0.beta3-dev and installing an AI plugin (limited to staff), Jose_C_Gomez experienced extreme site slowness and load errors despite normal server metrics. Initial investigations showed busy Unicorn workers, moderate crawler activity, and empty Sidekiq queues. pfaffman suggested checking for background tasks like image rebaking. simon flagged the NotifyMailingListSubscribers job as potentially resource-heavy and recommended checking error logs, which revealed timestamp overflow and IP spoofing errors — though likely unrelated. Jose_C_Gomez attempted to throttle crawlers via Discourse settings, but noted robots.txt didn’t reflect slow-crawler rules — clarified by supermathie that Discourse handles rate limiting internally. supermathie advised increasing Unicorn workers via app.yml and using the Prometheus plugin or performance headers for deeper diagnostics. Ultimately, Jose_C_Gomez resolved the issue by upgrading to a larger DigitalOcean instance (doubling CPU/RAM), increasing Unicorn workers to 8, and running database reindexing and vacuuming — restoring site performance.

1

Upgraded to 3.3.0.beta3-dev yesterday and also Installed AI plugin. The plugin is currently only enabled to staff member (5 people)

But the entire site is dog slow, I’m getting extreme load errors. I can’t seem to figure out where is it coming from my server load seems fine.

Is there a place or places I can go to figure out what is causing it.

2

Here’s what I see in the Crawler Report not sure if tis bad or good or what. I don’t have a frame of reference.

This image displays a line graph that shows the daily, weekly, and monthly web crawl pageview metrics over a period of time, along with options to view different date ranges and export or refresh the report. (Captioned by AI)
This image displays a line graph that shows the daily, weekly, and monthly web crawl pageview metrics over a period of time, along with options to view different date ranges and export or refresh the report. (Captioned by AI)1095×461 40 KB

3

Looking at my Server looks like unicorn processes are pretty busy

This image displays a terminal window showing system resource usage, including CPU, memory, and process information for various tasks such as PostgreSQL, nginx, and unicorn workers, which appear to be related to a web application or service. (Captioned by AI)
This image displays a terminal window showing system resource usage, including CPU, memory, and process information for various tasks such as PostgreSQL, nginx, and unicorn workers, which appear to be related to a web application or service. (Captioned by AI)1133×546 46.5 KB

Is this the cause? Do I need more CPU? or Just more Unicorns?

4

Has its been a while since your last upgrade? Maybe it’s doing some kind of image processing or rebaking.

You can have a look at /sidekiq to see what it’s doing.

5

Queues Are Empty

image
image1270×303 29.5 KB

I don’t really know what the rest of this means.

image
image1270×722 97 KB

image
image957×710 90.1 KB

I’m not sure what’s normal here… Here’s our server spec
image

6

Rebooted everything was back to normal but now we are getting extreme load again. I can’t figure out where the issue is coming from is there any tooling that can help within discourse?

7

So the 3 unicorn workers
image

Are busy… but we aren’t getting higher than normal traffic as far as I can tell it is about the same it has been the only change was upgrade to 3.3.0 and Added Ai plugin but its only available to Staff

Issues Started Yesterday 6/3

image
image1089×313 18.6 KB

We do have a few more crawlers it seems.

image
image844×324 26.1 KB

Here’s just Crawlers over a month, but again ti doesn’t seem that much higher. The site is almost un-usable

image
image1099×586 42.8 KB

Any help would be appreciated!

8

This is a guess, but the only thing that stands out for me in the Sidekiq logs is that the job that’s shown is NotifyMailingListSubscribers. That job can potentially create a lot of requests.

Also, do you see any errors on your Admin / Logs / Error Logs page?

9

I added a block to the facebook crawler cause that guy was going to town
image

However I noticed that adding slow / crawlers isn’t updating my robots.txt

image
image795×266 9.53 KB

but robots.txt doesn’t show the slow entries, only the block entries.

image
image773×832 12.3 KB

10

Quite a few of these

image
image755×106 5.62 KB

I see 3 errors but they don’t seem related… (though it is hard to tell)

Job exception: PG::DatetimeFieldOverflow: ERROR:  timestamp out of range: "271768-09-23 06:24:11.793040 BC"
LINE 1: ...sers"."moderator" = FALSE AND (users.created_at < '271768-09...
                                                             ^

ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="10.10.121.119" HTTP_X_FORWARDED_FOR="14.140.10.244, 14.140.10.244")
app/controllers/topics_controller.rb:1298:in `track_visit_to_topic'
app/controllers/topics_controller.rb:169:in `show'
app/controllers/application_controller.rb:422:in `block in with_resolved_locale'
app/controllers/application_controller.rb:422:in `with_resolved_locale'
lib/middleware/omniauth_bypass_middleware.rb:64:in `call'
lib/content_security_policy/middleware.rb:12:in `call'
lib/middleware/anonymous_cache.rb:391:in `call'
lib/middleware/csp_script_nonce_injector.rb:12:in `call'
config/initializers/008-rack-cors.rb:14:in `call'
config/initializers/100-quiet_logger.rb:20:in `call'
config/initializers/100-silence_logger.rb:29:in `call'
lib/middleware/enforce_hostname.rb:24:in `call'
lib/middleware/request_tracker.rb:291:in `call'

And another job exception around SMTP

11

Discourse does it’s own rate limiting, it doesn’t rely on robots.txt

12

Thanks Michael,

Any other ideas on what it could be? Would spinning more unicorns help?

Is that done form the app.yml?

13

Yes, that would probably help.

env:
  UNICORN_WORKERS: 8

in the app.yml will do this.

I recommend pulling performance numbers using the prometheus plugin if you have that set up, or you can use performance headers.

Analysing your web logs should help a lot with identifying why your server is so busy; looks like crawlers are a good place to start.

14

Well upgraded to a new DO instance doubled the ram and CPU. Added 8 unicorns (vs 3) did a Db reindex and vaccum and I think we are back in bussiness!

Thanks for the help