# Site goes down at the same time every day in memory-constrained environment
**URL:** https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358
**Category:** Support
**Created:** [December 18, 2017, 10:57pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358 "2017-12-18T22:57:22Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 10:57pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/1 "2017-12-18T22:57:22Z")
Hi,
I’m not sure what this could be, and I think running Discourse on 1GB of memory plus 1GB of swap is probably below minimum spec at this point so I’d understand if this is outright unsupported, but I’m having a weird issue where my site goes down at exactly the same time every day (at around 2:48 PST) for five minutes or so. Trying to hit it in the meantime sometimes throws 429’s.
sshing into the server and eyeballing top when this happens shows that about half of swap has been released (suggesting that maybe something else is running on the server and elbowing out Discourse, but I don’t know what it could be, there’s nothing in cron for that time of day), and most of the CPU is consumed by a single postmaster process until it runs for a little over two minutes, is killed, and then everything gradually goes back to normal.
Any idea what’s up here? Been happening for the past several releases at least.
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 11:04pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/2 "2017-12-18T23:04:42Z")
I don’t think it could be a crawler given that I don’t know any crawlers that are that punctual…
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [December 18, 2017, 11:06pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/3 "2017-12-18T23:06:31Z")
Per previous discussions it is almost certainly the database backup blowing out all your memory. Increase swap to 2GB.
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 11:10pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/4 "2017-12-18T23:10:25Z")
Will do, thanks! Probably want to update your install docs to reflect that if you haven’t already.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [December 18, 2017, 11:12pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/5 "2017-12-18T23:12:27Z")
This is a relatively new problem cc @sam
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 11:33pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/6 "2017-12-18T23:33:43Z")
For the record, I just checked that I have automatic backups disabled, unless it’s something else that e.g. redis does every day and isn’t exposed to the Discourse admin config. But I’ve increased swap anyway and we’ll see how tomorrow goes!
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [December 18, 2017, 11:40pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/7 "2017-12-18T23:40:27Z")
Hi Alex,
429 is a very odd response code for the site to be sending due to memory problems. That’s normally because someone on the same IP address is doing something untoward. Is there any chance someone else on the same connection (a person or machine in your office, for example) is running some sort of scraping batch job at the same time?
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 11:43pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/8 "2017-12-18T23:43:03Z")
Those might be a red herring – I’m running the container behind an nginx on the host so I can keep some old LAMP apps up on a couple other subdomains, and I think it just gets hit a lot by people trying to access the site when it goes down like this, with that being the result.
And I know I probably shouldn’t be doing that at or below minimum spec, so you’re free to disavow, but those haven’t changed in a dog’s age and this has only started happening with the most recent Discourse versions, always at the same time of day.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [December 18, 2017, 11:45pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/9 "2017-12-18T23:45:19Z")
> [@axfelix](#):
>
> and I think it just gets hit a lot by people trying to access the site when it goes down like this, with that being the result.
Well one thing I would check is that NGINX inside the container (that has the rate limiting template enabled) does not think every single user has the same IP address.
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 18, 2017, 11:49pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/10 "2017-12-18T23:49:25Z")
I don’t think the nginx _inside_ the container has diverged from stock at all, I use the provided launcher script for all the docker stuff.
This is what the host nginx config for the site looks like if you’re curious, but I don’t think this is the problem:
root@selectbutton:/etc/nginx/sites-enabled# cat discourse
```
server {
listen 443 ssl;
server_name selectbutton.net;
ssl_certificate /etc/letsencrypt/live/selectbutton.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/selectbutton.net/privkey.pem;
location /basic_status {
stub_status on;
access_log off;
allow 127.0.0.1;
allow 172.31.20.111;
allow 52.35.138.13;
deny all;
}
location / {
root /var/www/discourse-root;
try_files $uri @discourse;
}
location @discourse {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass https://localhost:4443;
}
}
server {
listen 80;
server_name selectbutton.net;
return 301 https://$host$request_uri;
}
```
---
### Author:  [@yanokwa](https://meta.discourse.org/u/yanokwa)
#### Post date: [December 19, 2017, 12:12am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/11 "2017-12-19T00:12:44Z")
I’m on a server with 2 GB of ram and I’m seeing the same behavior as of a few days ago. Seems to happen around backup time.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [December 19, 2017, 12:24am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/12 "2017-12-19T00:24:47Z")
> [@axfelix](#):
>
> I don’t think the nginx inside the container has diverged from stock at all, I use the provided launcher script for all the docker stuff.
Well you would be using this:
>
And if I had to bet a 💸 you do not have a template setting:
```plaintext
real_ip_header X-Forwarded-For;
```
The internal NGINX is going to have to trust that header.
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 19, 2017, 12:31am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/13 "2017-12-19T00:31:45Z")
Ah, OK, good to know. So I should add that to the host nginx or the container nginx (which is to say, that to: block in the app.yml?)
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [December 19, 2017, 12:35am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/14 "2017-12-19T00:35:42Z")
You need to run a replace command in the internal NGINX.
After making the changes, look at the actual file on disk to confirm that you got what you wanted.
A trick I use is:
1. ./launcher enter app
2. cd `/etc/nginx/conf.d/`
3. edit the discourse.conf file
4. sv restart nginx
5. see that my desired effect was achieved
6. turn that change into a yml `replace` command
7. rebuild
8. confirm file is good
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 19, 2017, 12:38am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/15 "2017-12-19T00:38:21Z")
OK, will test if it goes down again tomorrow despite increasing swap. Any other ideas if this turns out not to be it? The timeliness still makes me quite skeptical.
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [December 19, 2017, 12:43am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/16 "2017-12-19T00:43:05Z")
Yeah, the strict periodicity (in the absence of scheduled backups) is a very confusing aspect to this. Seems like a heck of a big clue, if you can figure out what’s running at that exact time, I think you’ll be 90% done.
---
### Author:  [@Dmitry\_Krasnoperov](https://meta.discourse.org/u/Dmitry_Krasnoperov)
#### Post date: [December 19, 2017, 8:48am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/17 "2017-12-19T08:48:09Z")
Same problem for me ☹
I’m tired of rebuild after every crash.
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [December 19, 2017, 8:57am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/18 "2017-12-19T08:57:22Z")
Check your `/sidekiq` and see if anything is running at that time.
What also might help is to start psql to take a look at what is going on in your database at that time.
`SELECT pid, age(query_start, clock_timestamp()), usename, query FROM pg_stat_activity;`
---
### Author:  [@Dmitry\_Krasnoperov](https://meta.discourse.org/u/Dmitry_Krasnoperov)
#### Post date: [December 19, 2017, 9:25am UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/19 "2017-12-19T09:25:43Z")
> [@RGJ](#):
>
> Check your /sidekiq and see if anything is running at that time.
Sidekiq is idle 🤔
---
### Author:  [@axfelix](https://meta.discourse.org/u/axfelix)
#### Post date: [December 19, 2017, 10:51pm UTC](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358/20 "2017-12-19T22:51:35Z")
And of course it seems not to have happened today after occurring several days in a row at this exact time. Huh. swap usage hasn’t ticked over 900M either. Oh well! Stay tuned I guess.
[Next page](https://meta.discourse.org/t/site-goes-down-at-the-same-time-every-day-in-memory-constrained-environment/76358.md?page=2)