Hi, I’m not sure what this could be, and I think running Discourse on 1GB of memory plus 1GB of swap is probably below minimum spec at this point so I’d understand if this is outright unsupported, but I’m having a weird issue where my site goes down at exactly the same time every day (at around 2:48 PST) for five minutes or so. Trying to hit it in the meantime sometimes throws 429’s. sshing into the server and eyeballing top when this happens shows that about half of swap has been released (suggesting that maybe something else is running on the server and elbowing out Discourse, but I don’t know what it could be, there’s nothing in cron for that time of day), and most of the CPU is consumed by a single postmaster process until it runs for a little over two minutes, is killed, and then everything gradually goes back to normal. Any idea what’s up here? Been happening for the past several releases at least.

I’m on a server with 2 GB of ram and I’m seeing the same behavior as of a few days ago. Seems to happen around backup time.

[image] axfelix: I don’t think the nginx inside the container has diverged from stock at all, I use the provided launcher script for all the docker stuff. Well you would be using this: templates/web.ratelimited.template.yml master params: reqs_per_second: 12 burst_per_second: 12 reqs_per_minute: 200 burst_per_minute: 100 conn_per_ip: 20 run: - replace: filename: "/etc/nginx/conf.d/discourse.conf" from: /server.+{/ to: | limit_req_zone $binary_remote_addr zone=flood:10m rate=$reqs_per_secondr/s; limit_req_zone $binary_remote_addr zone=bot:10m rate=$reqs_per_minuter/m; limit_req_status 429; limit_conn_zone $binary_remote_addr zone=connperip:10m; limit_conn_status 429; server { - replace: filename: "/etc/nginx/conf.d/discourse.conf" This file has been truncated. show original And if I had to bet a :money_with_wings: you do not have a template setting: real_ip_header X-Forwarded-For; The internal NGINX is going to have to trust that header.

Ah, OK, good to know. So I should add that to the host nginx or the container nginx (which is to say, that to: block in the app.yml?)

You need to run a replace command in the internal NGINX. After making the changes, look at the actual file on disk to confirm that you got what you wanted. A trick I use is: ./launcher enter app cd /etc/nginx/conf.d/ edit the discourse.conf file sv restart nginx see that my desired effect was achieved turn that change into a yml replace command rebuild confirm file is good

OK, will test if it goes down again tomorrow despite increasing swap. Any other ideas if this turns out not to be it? The timeliness still makes me quite skeptical.

Yeah, the strict periodicity (in the absence of scheduled backups) is a very confusing aspect to this. Seems like a heck of a big clue, if you can figure out what’s running at that exact time, I think you’ll be 90% done.

Same problem for me :frowning: I’m tired of rebuild after every crash.

Check your /sidekiq and see if anything is running at that time. What also might help is to start psql to take a look at what is going on in your database at that time. SELECT pid, age(query_start, clock_timestamp()), usename, query FROM pg_stat_activity;

[image] RGJ: Check your /sidekiq and see if anything is running at that time. Sidekiq is idle :thinking:

And of course it seems not to have happened today after occurring several days in a row at this exact time. Huh. swap usage hasn’t ticked over 900M either. Oh well! Stay tuned I guess.

Site goes down at the same time every day in memory-constrained environment

Product Support

axfelix (Alex Garnett) December 19, 2017, 11:37pm 22

It’s not always a 429 – throws 408’s too.

Topic		Replies	Views
429 too many requests good or bad ? (please help) Self-hosting	14	423	August 25, 2026
Is 1GB RAM enough for both WordPress and Discourse? Self-hosting server-resources	21	5061	March 3, 2017
Bursts of 502 Service Unavailable, pointers to debug Self-hosting	15	1839	February 18, 2018
Troubleshooting a 429 (rate limit) Self-hosting	19	6332	October 10, 2018
Discourse overloaded real traffic or DDOS? 100% CPU usage despite of decent traffic and high specs server Self-hosting server-resources	18	2675	September 25, 2021

Site goes down at the same time every day in memory-constrained environment

Related topics