Nginx upstream timed out (110: Connection timed out)

Hi my site is https://1o24bbs.com/

No upgrade no operation, out of nowhere nginx shows it fails to connect to upstream: http://127.0.0.1:3000

How do I trouble shoot it? Appreciate the help

/shared/log/rails# tail -f unicorn.stdout.log

2021-07-08T09:01:03.403Z pid=682 tid=sim ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

2021-07-08T09:01:03.412Z pid=682 tid=t4m ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

2021-07-08T09:01:03.406Z pid=682 tid=tra ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

2021-07-08T09:01:03.407Z pid=682 tid=sn6 ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

2021-07-08T09:01:03.407Z pid=682 tid=sze ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

Loading Sidekiq in process id 648

Kill self supervisor is gone

Kill self supervisor is gone

Kill self supervisor is gone

Loading Sidekiq in process id 780

/var/log/redis# tail -f current

553:M 08 Jul 2021 09:11:59.078 * 10000 changes in 60 seconds. Saving…

553:M 08 Jul 2021 09:11:59.079 * Background saving started by pid 1270

1270:C 08 Jul 2021 09:12:01.584 * DB saved on disk

1270:C 08 Jul 2021 09:12:01.595 * RDB: 0 MB of memory used by copy-on-write

553:M 08 Jul 2021 09:12:01.619 * Background saving terminated with success

553:M 08 Jul 2021 09:13:02.098 * 10000 changes in 60 seconds. Saving…

553:M 08 Jul 2021 09:13:02.099 * Background saving started by pid 1339

1339:C 08 Jul 2021 09:13:04.052 * DB saved on disk

1339:C 08 Jul 2021 09:13:04.055 * RDB: 0 MB of memory used by copy-on-write

553:M 08 Jul 2021 09:13:04.137 * Background saving terminated with success

During rebuild app, there is error Failed listening on port 6379

Is this a Discourse official Standard Installation?

Do you have sufficient disk space (I guess you’d have noticed if you tried to rebuild).

It appears to be a problem with redis, though I can’t tell why from what you’ve put here.

Yes, the website has been running fine for years

I am beginning to think maybe it is not redis issue.
Sidekiq is more of a background job, right?

That should not make http://127.0.0.1:3000 not reachable

More logs in nginx

2021/07/08 10:54:31 [error] 570#570: *169 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/4cf07f106ff14fd885fd735ea28abce4/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/4cf07f106ff14fd885fd735ea28abce4/poll?dlp=t”, host: “1o24bbs.com”, referrer: “https://1o24bbs.com/t/icc/25834

2021/07/08 10:54:32 [error] 570#570: *169 upstream timed out (110: Connection timed out) while connecting to upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/fe81f75495324a109de3812a3c5a9abd/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/fe81f75495324a109de3812a3c5a9abd/poll?dlp=t”, host: “1o24bbs.com”, referrer: “https://1o24bbs.com/t/icc/23914

2021/07/08 10:54:33 [error] 570#570: *169 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/1dd105462c5b4b4c978e1702fa885e14/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/1dd105462c5b4b4c978e1702fa885e14/poll?dlp=t”, host: “1o24bbs.com”, referrer: “https://1o24bbs.com/t/topic/15436

2021/07/08 10:54:36 [error] 570#570: *13261 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 220.196.56.24, server: _, request: “GET /?374e828961d36d46bd99dc8e13b7f640 HTTP/1.1”, upstream: “http://127.0.0.1:3000/?374e828961d36d46bd99dc8e13b7f640”, host: “1o24bbs.com”, referrer: “https://gateway.zscalertwo.net:443/auT?origurl=https%3A%2F%2F1o24bbs.com%2F%3F374e828961d36d46bd99dc8e13b7f640&wexps=1&_ordtok=SWk3WVhmhHBPJ4rLFk4nDRnkZ3&wexps=1

In production.log, I see

Processing by CategoriesController#index as HTML
Parameters: {“41528430213d6ae753e074e11328b67b”=>nil}
Job exception: FATAL: the database system is shutting down

@pfaffman How do I flush redis? Thanks!

Have you done PostgreSQL 13 update.

Thanks!
My website is running the latest version of discourse. I have been running the latest version for at least one month.

I didn’t manually upgrade PogreSQL but I think Postgresql is version 13.
How do I make sure of that?

$ psql -V
psql (PostgreSQL) 13.3 (Debian 13.3-1.pgdg100+1)

BTW, I am running a single container app

Do you have any non-standard plugins?

Plugins I have:

  • docker_manager
  • discourse-adplugin

Do you think the issue is with redis or postgresql?

After # ./launcher enter app when I run curl http://127.0.0.1:3000 it times out

Output of discourse-doctor:

==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 19.03.13, build 4484c46d9d

DOCKER PROCESSES (docker ps -a)

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                   PORTS                                      NAMES
70709beca56d        local_discourse/app                "/sbin/boot"             15 minutes ago      Up 15 minutes            0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   app
1828993cd957        discourse/base:2.0.20210528-1735   "/bin/bash -c 'cd /p…"   2 hours ago         Exited (1) 2 hours ago                                              agitated_curie

70709beca56d        local_discourse/app   "/sbin/boot"        15 minutes ago      Up 15 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   app

Discourse container app is running


==================== PLUGINS ====================
          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-adplugin.git

No non-official plugins detected.

See https://github.com/discourse/discourse/blob/master/lib/plugin/metadata.rb for the official list.

========================================

Sorry. I don’t have any ideas. Have you rebooted lately?

I had to bring up a new VM and use db backup to create a new container
But it seems to be happening again after that …

had the same problem with a company website foreign exchange send money online, tried to rebot the server and change the timeout to 1000 sec and then try it out.