# Bursts of 502 Service Unavailable, pointers to debug
**URL:** https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571
**Category:** Self-hosting
**Created:** [2월 14, 2018, 9:10오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571 "2018-02-14T21:10:12Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 14, 2018, 9:10오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/1 "2018-02-14T21:10:12Z")
Hi,
I run [discuss.pytorch.org](http://discuss.pytorch.org) hosted on DigitalOcean. A couple of nights ago, the website became REALLY slow, and we were seeing “nginx 502 service unavailable” pages.
Disk space was plenty available, nothing changed recently.
`htop` showed that there was enough processing available, cores were not fully occupied.
The only thing I remember changing on the machine in a recent time window was that I did a `apt dist-upgrade` a week earlier to get Spectre/Meltdown patches.
After looking around on the forums and finding no references / pointers that are relevant, I decided to take a backup, and then nuked the machine.
I created a fresh machine, updated 16.04 to the latest kernel / patches, and deployed a new discourse install. I t was REALLY slow / was 502-ing. I presumed that the upgrades were probably the issue.
So, I created yet another fresh machine, didn’t update 16.04 and deployed a new discourse install. Then I restored from backup. The site was back to normal performance.
However, today afternoon for a period of an hour or so, users saw 502 Service Unavailable, and then the site came back to normal performance.
**My question is:**
- can I get any pointers to handle such situations, from a sysadmin perspective? What exactly to look for, any log parsing.
- are there any recent known issues with discourse installs across the community with elevated 502 Service Unavailable errors?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2월 15, 2018, 2:34오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/2 "2018-02-15T02:34:22Z")
Sounds to me like out of memory.
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 15, 2018, 3:21오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/3 "2018-02-15T03:21:28Z")
sorry I should’ve mentioned that I also monitored memory. Memory is plenty available (50% of memory occupied). I got the 4 CPU, 8GB Mem 160GB Disk instance, and it almost never goes to peak on any of the resources.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2월 15, 2018, 11:29오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/4 "2018-02-15T23:29:42Z")
> [@smth](#):
>
> The only thing I remember changing on the machine in a recent time window was that I did a apt dist-upgrade a week earlier to get Spectre/Meltdown patches.
This is going to cost you in performance, its unavoidable.
What version of Discourse are you running? tests-passed branch?
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 16, 2018, 4:12오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/5 "2018-02-16T16:12:56Z")
I am running whatever came (at that time) with the autoupdater, i.e. [v2.0.0.beta2 +144](https://github.com/discourse/discourse/compare/03b3e57a44da228bca7296bd752e26447956e1d6...tests-passed) – tests-passed.
The flakiness hasn’t repeated since, so I think the thread is good to close.
But having any pointers on what you guys (as the core-devs) would go hunt down in terms of log files or other things in such situations would be helpful, so that I can bother you less and become self-sufficient 🙂
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2월 17, 2018, 12:15오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/6 "2018-02-17T00:15:02Z")
Any ideas of where to look @mpalmer? In a general sense? There might be a good post you wrote on this on meta, somewhere?
---
### Author:  [@mpalmer](https://meta.discourse.org/u/mpalmer)
#### Post date: [2월 17, 2018, 1:03오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/7 "2018-02-17T01:03:28Z")
I can’t think of anything I’ve written that covers tracing a request through the layers. I’d suggest starting with @Sam’s topic on [where the wild logs are](https://meta.discourse.org/t/where-are-all-the-discourse-logs/58022) and working from the outside in – find a failing request in the logs for the outermost proxy (either nginx in the container, or the host’s nginx/apache if it’s setup that way), then find the corresponding entry at the next hop (unicorn, or container nginx), and so on.
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 17, 2018, 7:44오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/8 "2018-02-17T19:44:41Z")
Thanks a lot. I’ve been seeing 502 again, and I see full CPU usage. I’ll start following the wild logs and see how far I get.
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 17, 2018, 8:06오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/9 "2018-02-17T20:06:22Z")
The post with the pointer to logs is super helpful.
Lots of workers are timing out:
unicorn.stderr.log
```plaintext
E, [2018-02-17T19:44:21.208533 #800] ERROR -- : worker=4 PID:29923 timeout (31s > 30s), killing
I, [2018-02-17T19:44:29.211582 #29979] INFO -- : worker=5 ready
I, [2018-02-17T19:44:29.699253 #29990] INFO -- : worker=7 ready
I, [2018-02-17T19:44:31.985875 #30008] INFO -- : worker=2 ready
I, [2018-02-17T19:44:46.317249 #30050] INFO -- : worker=4 ready
E, [2018-02-17T19:45:57.690129 #800] ERROR -- : worker=5 PID:29979 timeout (36s > 30s), killing
E, [2018-02-17T19:46:07.212510 #800] ERROR -- : worker=3 PID:29952 timeout (35s > 30s), killing
E, [2018-02-17T19:46:07.221215 #800] ERROR -- : worker=7 PID:29990 timeout (33s > 30s), killing
E, [2018-02-17T19:46:07.783812 #800] ERROR -- : worker=7 PID:29990 timeout (33s > 30s), killing
I, [2018-02-17T19:46:44.135563 #30165] INFO -- : worker=3 ready
I, [2018-02-17T19:46:44.840466 #30156] INFO -- : worker=5 ready
I, [2018-02-17T19:46:46.952119 #30174] INFO -- : worker=7 ready
E, [2018-02-17T19:46:59.823120 #800] ERROR -- : worker=2 PID:30008 timeout (31s > 30s), killing
E, [2018-02-17T19:47:43.274893 #800] ERROR -- : worker=7 PID:30174 timeout (31s > 30s), killing
E, [2018-02-17T19:47:46.393225 #800] ERROR -- : worker=1 PID:29767 timeout (33s > 30s), killing
E, [2018-02-17T19:47:46.579485 #800] ERROR -- : worker=5 PID:30156 timeout (31s > 30s), killing
I, [2018-02-17T19:47:46.686074 #30254] INFO -- : worker=2 ready
E, [2018-02-17T19:47:46.894589 #800] ERROR -- : worker=1 PID:29767 timeout (33s > 30s), killing
E, [2018-02-17T19:47:49.736359 #800] ERROR -- : worker=3 PID:30165 timeout (32s > 30s), killing
I, [2018-02-17T19:48:14.864398 #30307] INFO -- : worker=5 ready
I, [2018-02-17T19:48:16.177603 #30336] INFO -- : worker=3 ready
I, [2018-02-17T19:48:16.345038 #30322] INFO -- : worker=1 ready
I, [2018-02-17T19:48:20.622346 #30296] INFO -- : worker=7 ready
D, [2018-02-17T19:48:22.908873 #800] DEBUG -- : waiting 16.0s after suspend/hibernation
E, [2018-02-17T19:49:22.238502 #800] ERROR -- : worker=5 PID:30307 timeout (31s > 30s), killing
E, [2018-02-17T19:49:31.480543 #800] ERROR -- : worker=4 PID:30050 timeout (32s > 30s), killing
E, [2018-02-17T19:49:31.588904 #800] ERROR -- : worker=1 PID:30322 timeout (33s > 30s), killing
E, [2018-02-17T19:49:32.221465 #800] ERROR -- : worker=1 PID:30322 timeout (34s > 30s), killing
E, [2018-02-17T19:49:45.783025 #800] ERROR -- : worker=7 PID:30296 timeout (32s > 30s), killing
E, [2018-02-17T19:49:50.598973 #800] ERROR -- : worker=3 PID:30336 timeout (31s > 30s), killing
```
After digging in, it looks like postgres is being hammered, with a lot of queries taking 700+ms. one query took 50 seconds.
I’ve posted the last 1000 lines here: [https://gist.github.com/886f2324e50958dea43c51a2595ec15d](https://gist.github.com/886f2324e50958dea43c51a2595ec15d)
Edit: deleted log – security
If anything looks weird or non-standard, I’d appreciate if anyone lets me know.
For now I’ll try upgrading to an even more beefier instance and see if that helps.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2월 17, 2018, 9:27오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/10 "2018-02-17T21:27:55Z")
Can you quote the stats on your /about page? Did you migrate data from another forum software? Seems like there is something strange about your database.
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 17, 2018, 9:42오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/11 "2018-02-17T21:42:37Z")
> Did you migrate data from another forum software?
No. Started fresh with Discourse and nothing else.
Here are stats:
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2월 18, 2018, 5:43오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/12 "2018-02-18T05:43:25Z")
I am going to need to see the sql log
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 18, 2018, 6:44오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/13 "2018-02-18T06:44:33Z")
@sam thanks for wanting to see the sql logs. I sent you a message with them.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2월 18, 2018, 7:15오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/14 "2018-02-18T07:15:42Z")
I had a look at the log there, nothing sticks out, just looks like server is totally overloaded
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2월 18, 2018, 7:43오전 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/15 "2018-02-18T07:43:13Z")
Looking at that screenshot, the high CPU is coming from the app. unicorns and sidekiq are all very high there. You can see that even htop is taking tons of CPU in that case. My guess is that you have some very bursty traffic headed your way.
Also looking at your machine running 7/8 unicorns seems overkill, I would maybe run 5 in your current setup otherwise there is very high risk of a thundering herd taking everyone down.
Can you include the expanded screenshot of detailed traffic reports? (bottom of dashboard)
---
### Author:  [@smth](https://meta.discourse.org/u/smth)
#### Post date: [2월 18, 2018, 4:37오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/16 "2018-02-18T16:37:33Z")
Yesterday, I upgraded to a VERY beefy instance, CPU Optimized Droplets with 16 vCPUs. (previously Standard Droplet with 4 vCPUs)
It’s an overkill for the workload, but I wanted to first upgrade and then gradually scale down. It’s working extremely well.
And @sam you were right about bursty traffic. Looking at the traffic logs, I see massive spike in server errors right around the crashes.
**[the site was recovered from backup mid-week last week to see if it was the instance-level issues, so traffic report before that might be blank before that date]**
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [6월 8, 2024, 12:37오후 UTC](https://meta.discourse.org/t/bursts-of-502-service-unavailable-pointers-to-debug/80571/17 "2024-06-08T12:37:48Z")
이 주제는 2305일 후 자동으로 닫혔습니다. 더 이상 새로운 답변을 남길 수 없습니다.