How to diagnose request timeouts


(David Mitchell) #1

I have an installation of discourse that started responding with 504’s for every request this afternoon. After digging into it a bit, I discovered that unicorn was shutting down child workers after timeouts.

I’ve tried a number of things: rebuilding the container, restarting the host machine and the database server, and checking every log that I can find, but I’ve been unable to locate anything that seems like the source of the issue. I turned the log_level for the Ruby app all the way up to :debug, but all I can see are the timings for SQL statements (which seem fine).

What else can I do to diagnose this issue?


(Sam Saffron) #2

Sounds t me likely that either DB or redis are choking.

How is memory and disk space on the box? With redis you can try a flushall to release some extra memory.


(David Mitchell) #3

I performed a flushall on Redis, and restarted the system that the DB lives on as well as the system hosting Discourse. (the DB server, web server, and Redis server are on discrete systems).

The site has now mysteriously recovered. If it occurs again, I’ll be sure to check on free memory and disk on the servers.

Are there any other things I can check if this occurs again?


(Sam Saffron) #4

What is disk space and memory like? Is there anything in /logs?


(David Mitchell) #5

The only log messages I could find were from Unicorn shutting down processes after they had timed out. I checked Postgres during the outage, and there were no long-running queries; most of the connections were idle. Redis showed more activity, but it seemed to be humming along just fine.

Here are memory/disk stats for the nodes I’m using

Web

Memory

MemTotal:        4049156 kB
MemFree:         1266652 kB

Disk

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   20G   18G  53% /

Postgres

Memory

MemTotal:        4049940 kB
MemFree:         3749472 kB

Disk

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G  1.5G   36G   5% /

Redis

Memory

MemTotal:        4049940 kB
MemFree:         2834276 kB

Disk

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G  2.3G   36G   7% /