Grootmade@app-node - 11: Resource temporarily unavailable

Support Self-hosting
1

Hey everyone,

I’m hitting a wall with an intermittent connection-dropping issue on our production Discourse app node and need some advice from sysadmins or kernel networking experts.

During traffic spikes, Nginx starts returning 502 Bad Gateway and dropped connections cascade across upstream services. At first, I assumed it was a standard Postgres/Redis pool limit, but checking the network stack points toward socket exhaustion and conntrack table saturation.

What I’ve Verified

  • Limits are bumped to 65535 in limits.conf and verified for running processes.
  • Puma/Sidekiq queues are normal, CPU utilization is hovering at ~45%.
  • Outbound API calls and incoming user handshakes hang intermittently for 10–30 seconds.

Below is the raw terminal history from my session right after the last spike, showing what I checked and the errors thrown:

grootmade@app-node-01:~$ netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
      4 LAST_ACK
     12 LISTEN
     48 FIN_WAIT2
    312 ESTABLISHED
   8940 TIME_WAIT

grootmade@app-node-01:~$ sysctl net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_reuse = 2

grootmade@app-node-01:~$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262140
net.netfilter.nf_conntrack_max = 262144

grootmade@app-node-01:~$ dmesg -T | grep -i conntrack | tail -n 5
[Sun Aug 23 14:10:02 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Sun Aug 23 14:10:05 2026] nf_conntrack: nf_conntrack: table full, dropping packet
[Sun Aug 23 14:10:12 2026] nf_conntrack: nf_conntrack: table full, dropping packet

grootmade@app-node-01:~$ sudo ss -s
Total: 9648
TCP:   9410 (estab 312, closed 8940, orphaned 0, timewait 8940)

grootmade@app-node-01:~$ cat /proc/sys/net/ipv4/ip_local_port_range
32768	60999

grootmade@app-node-01:~$ ulimit -n
65535

grootmade@app-node-01:~$ cat /proc/sys/fs/file-nr
4128	0	65535

grootmade@app-node-01:~$ sudo iptables -L -n -v | grep -i DROP
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

grootmade@app-node-01:~$ sudo strace -p 1842 -e trace=connect,bind 2>&1 | head -n 10
connect(14, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.1.25")}, 16) = -1 EAGAIN (Resource temporarily unavailable)

grootmade@app-node-01:~$ cat /etc/security/limits.d/discourse.conf
discourse soft nofile 65535
discourse hard nofile 65535

grootmade@app-node-01:~$ tail -n 20 /var/log/nginx/error.log | grep "failed"
2026/08/23 14:10:15 [crit] 1842#1842: *49102 connect() to 10.0.1.25:5432 failed (11: Resource temporarily unavailable)

grootmade@app-node-01:~$ history | tail -n 13

My Questions:

  1. What are the recommended memory trade-offs for scaling nf_conntrack_max to 1048576 on a 16GB RAM node?

  2. Even with tcp_tw_reuse = 2, we are getting EAGAIN on local port bindings to Postgres. Should we expand ip_local_port_range or enable HTTP keep-alives upstream in Nginx?

  3. Would adding NOTRACK raw iptables rules for internal loopback and VPC traffic be a safe permanent fix, or does it introduce hidden state issues with container networks?

Any advice or battle-tested sysctl.conf snippets would be deeply appreciated!

Thanks!

2

Modern Discourse does not run Puma. What version of discourse are you running? Is it fair to say this is a non-standard install?