My Discourse is doing great, but I’ve been checking my logs and sometimes the connection times out. I know this is not a Discourse problem per se, and more of a Docker one, but it might be useful for newbies (like me) to know how to tune our Nginx, Docker or Discourse to avoid this. I already removed the default “fail_timeout”. Not sure if it did anything though. Any help is much appreciated :slight_smile: Here’s my log: 2015/11/09 19:32:15 [error] 3094#0: *61192 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 45.30.160.151, server: nomadforum.io, request: "POST /message-bus/3be12904dd624ddc83b6adba28cca586/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/3be12904dd624ddc83b6adba28cca586/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/introduce-yourself-who-are-you-where-are-you-and-what-do-you-do/229/2" 2015/11/09 19:32:16 [error] 3094#0: *71793 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 182.150.160.50, server: nomadforum.io, request: "POST /message-bus/5299d452871c45d198a935c65929d9e0/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/5299d452871c45d198a935c65929d9e0/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/whats-a-great-place-for-remote-work-and-self-realization-in-se-asia/3108" 2015/11/09 19:32:16 [error] 3095#0: *69594 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 61.90.15.118, server: nomadforum.io, request: "POST /message-bus/c292dc35ed0b4096a52236274383618e/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/c292dc35ed0b4096a52236274383618e/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/how-do-you-find-love-as-a-digital-nomad/5032/2" 2015/11/09 19:32:27 [error] 3098#0: *59372 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 125.255.46.197, server: nomadforum.io, request: "POST /message-bus/b21f969688714741b890cafd30d59fde/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/b21f969688714741b890cafd30d59fde/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/if-you-dont-have-a-residence-where-do-you-pay-taxes-as-a-european-citizen/1746/22" 2015/11/09 19:32:28 [error] 3095#0: *69594 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 61.90.15.118, server: nomadforum.io, request: "POST /message-bus/c292dc35ed0b4096a52236274383618e/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/c292dc35ed0b4096a52236274383618e/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/how-do-you-find-love-as-a-digital-nomad/5032/2" 2015/11/09 19:32:41 [error] 3094#0: *71793 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 182.150.160.50, server: nomadforum.io, request: "POST /message-bus/15040174a37442a9a5fb0095d8f64f2d/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/15040174a37442a9a5fb0095d8f64f2d/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/where-should-i-register-my-company-when-operating-in-multiple-countries-in-se-asia/1162" 2015/11/09 19:32:43 [error] 3095#0: *69594 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 61.90.15.118, server: nomadforum.io, request: "POST /message-bus/c292dc35ed0b4096a52236274383618e/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/c292dc35ed0b4096a52236274383618e/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/how-do-you-find-love-as-a-digital-nomad/5032/2" 2015/11/09 19:32:44 [error] 3094#0: *74158 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 202.151.194.72, server: nomadforum.io, request: "POST /message-bus/0b5419c26bd6485aab0e3af9b7008572/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/0b5419c26bd6485aab0e3af9b7008572/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/where-can-i-find-public-places-with-good-internet-to-work-in-kuala-lumpur/1302/6" 2015/11/09 19:32:49 [error] 3099#0: *75627 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.27.99.35, server: nomadforum.io, request: "POST /message-bus/4e0ae92b50554232be0432c7dc421710/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/4e0ae92b50554232be0432c7dc421710/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/whats-the-best-mobile-data-option-in-thailand-for-nomads/1193" 2015/11/09 19:32:58 [error] 3098#0: *75723 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 184.163.246.172, server: nomadforum.io, request: "POST /message-bus/679f981ab5f0459584b5bbd22416117c/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/679f981ab5f0459584b5bbd22416117c/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/best-place-to-stay-in-bangkok/1564" 2015/11/09 19:33:01 [error] 3095#0: *69594 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 61.90.15.118, server: nomadforum.io, request: "POST /message-bus/c292dc35ed0b4096a52236274383618e/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/c292dc35ed0b4096a52236274383618e/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/how-do-you-find-love-as-a-digital-nomad/5032/2" 2015/11/09 19:33:03 [error] 3094#0: *71793 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 182.150.160.50, server: nomadforum.io, request: "POST /message-bus/69bda49b64294b1aba909d50dd9df48b/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/69bda49b64294b1aba909d50dd9df48b/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/what-are-good-neighborhoods-in-chiang-mai/4791" 2015/11/09 19:33:04 [error] 3099#0: *75627 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 12.27.99.35, server: nomadforum.io, request: "POST /message-bus/4e0ae92b50554232be0432c7dc421710/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/4e0ae92b50554232be0432c7dc421710/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/whats-the-best-mobile-data-option-in-thailand-for-nomads/1193" 2015/11/09 19:33:13 [error] 3098#0: *75723 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 184.163.246.172, server: nomadforum.io, request: "POST /message-bus/679f981ab5f0459584b5bbd22416117c/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/679f981ab5f0459584b5bbd22416117c/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/best-place-to-stay-in-bangkok/1564" 2015/11/09 19:33:15 [error] 3094#0: *61192 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 45.30.160.151, server: nomadforum.io, request: "POST /message-bus/3be12904dd624ddc83b6adba28cca586/poll? HTTP/1.1", upstream: "http://127.0.0.1:4578/message-bus/3be12904dd624ddc83b6adba28cca586/poll?", host: "nomadforum.io", referrer: "https://nomadforum.io/t/introduce-yourself-who-are-you-where-are-you-and-what-do-you-do/229/2"

[תמונה] codinghorror: Not sure message bus polling timeouts really matter though. Message bus is used to poll for updates (e.g. new posts, threads) and update the page?

Yes but a “failure” there would never be visible to the user, and may not matter. Since the bus is always active, more messages will arrive later…

Based on your feedback I added a line to clarify that Unicorns should scale with CPU cores, when possible ## TODO: How many concurrent web requests are supported? ## With 2GB we recommend 3-4 workers, with 1GB only 2 ## If you have lots of memory, use one or two workers per logical CPU core #UNICORN_WORKERS: 3

(This is an old topic, but it still seems like where this belongs) [תמונה] codinghorror: one worker per CPU core (real or logical). I would allow at least 512mb ram per worker So the “ram per worker” is db_work_mem? That seems right, but the comments there don’t seem to say what you said above. ## Set higher on large instances it defaults to 10MB, for a 3GB install 40MB is a good default ## this improves sorting performance, but adds memory usage per-connection db_work_mem: "512MB"

[תמונה] work_mem is used several times per query. So a complex query (like suggested topics) can use 5~10 times that, and this for each query running and you are fast into swapping territory. The allow memory per work that Jeff cited, is more on don’t set 10 unicorn workers if you have a 10CPU server with 1GB ram.

You’re back! OK, I get that I’m wrong. Is this more on target? so with 8CPUs and 12GB db_work_mem: 40MB UNICORN_WORKERS: 8 and with 4 CPUs and 8GB db_work_mem: 40MB UNICORN_WORKERS: 4 And then I think also involved is the size of the database itself, which we want to be able to get buffered. With a huge database (say postgres_data is 8GB of disk space and there’s 8GB of ram), you might want to reduce the memory for the unicorn workers?

[תמונה] pfaffman: You’re back! Packing my stuff and took a minute to check Meta :blush: [תמונה] pfaffman: so with 8CPUs and 12GB db_work_mem: 40MB UNICORN_WORKERS: 8 and with 4 CPUs and 8GB db_work_mem: 40MB UNICORN_WORKERS: 4 Sounds good! [תמונה] pfaffman: With a huge database (say postgres_data is 8GB of disk space and there’s 8GB of ram) At this size, I would split machines: DB Machine Web Machine(s) Then you can check timings (the awesome mini_profiler info should be enough) and scale machines independently.

db_work_mem: 40MB UNICORN_WORKERS: 8 and with 4 CPUs and 8GB db_work_mem: 40MB UNICORN_WORKERS: 4 Hi @Falco, If we traspolate those values to use Puma, what is the recommended number of Threads per worker you suggest? In the case we are talking about a site with 1M requests per day. Cheers, Ismael

Is this a site that your are looking at moving to discourse? For 1M/day you may need to move to a more complex setup with haproxy or traefik reverse proxying to multiple backends (which is not supported here). How big is your database?

You can do multiple backend with nginx as well and it is well within the spec. Just serve static via a CDN

How to avoid upstream timeouts?

Product Support

sam (Sam Saffron) 25 בפברואר,‏ 2019,‏ 4:05am 25

NGINX Unit is not going to work properly until they implement this:

https://github.com/nginx/unit/issues/124

נושא		תגובות	צפיות
Due to extreme load, this is temporarily being shown to everyone... when it's not really the case Self-hosting server-resources	19	2063	21 ביולי,‏ 2023
Recommendation for number of workers: cores × 2? Self-hosting	16	768	29 באפריל,‏ 2025
High CPU usage (Ruby) Product Support server-resources	20	1141	3 בפברואר,‏ 2025
Best configurations for speeding up standalone discourse Self-hosting server-resources	25	4757	14 בפברואר,‏ 2024
502 Bad Gateway for only two Topics Self-hosting arm	19	2323	17 בנובמבר,‏ 2022

How to avoid upstream timeouts?

נושאים קשורים