Hi my site is https://1o24bbs.com/
No upgrade no operation, out of nowhere nginx shows it fails to connect to upstream: http://127.0.0.1:3000
How do I trouble shoot it? Appreciate the help
/shared/log/rails# tail -f unicorn.stdout.log
2021-07-08T09:01:03.403Z pid=682 tid=sim ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2021-07-08T09:01:03.412Z pid=682 tid=t4m ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2021-07-08T09:01:03.406Z pid=682 tid=tra ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2021-07-08T09:01:03.407Z pid=682 tid=sn6 ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2021-07-08T09:01:03.407Z pid=682 tid=sze ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
Loading Sidekiq in process id 648
Kill self supervisor is gone
Kill self supervisor is gone
Kill self supervisor is gone
Loading Sidekiq in process id 780
/var/log/redis# tail -f current
553:M 08 Jul 2021 09:11:59.078 * 10000 changes in 60 seconds. Saving…
553:M 08 Jul 2021 09:11:59.079 * Background saving started by pid 1270
1270:C 08 Jul 2021 09:12:01.584 * DB saved on disk
1270:C 08 Jul 2021 09:12:01.595 * RDB: 0 MB of memory used by copy-on-write
553:M 08 Jul 2021 09:12:01.619 * Background saving terminated with success
553:M 08 Jul 2021 09:13:02.098 * 10000 changes in 60 seconds. Saving…
553:M 08 Jul 2021 09:13:02.099 * Background saving started by pid 1339
1339:C 08 Jul 2021 09:13:04.052 * DB saved on disk
1339:C 08 Jul 2021 09:13:04.055 * RDB: 0 MB of memory used by copy-on-write
553:M 08 Jul 2021 09:13:04.137 * Background saving terminated with success
During rebuild app, there is error Failed listening on port 6379
Is this a Discourse official Standard Installation?
Do you have sufficient disk space (I guess you’d have noticed if you tried to rebuild).
It appears to be a problem with redis, though I can’t tell why from what you’ve put here.
Yes, the website has been running fine for years
I am beginning to think maybe it is not redis issue.
Sidekiq is more of a background job, right?
That should not make http://127.0.0.1:3000 not reachable
More logs in nginx
2021/07/08 10:54:31 [error] 570#570: *169 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/4cf07f106ff14fd885fd735ea28abce4/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/4cf07f106ff14fd885fd735ea28abce4/poll?dlp=t”, host: “1o24bbs.com”, referrer: “求ICC的群 - CS News - 1024 BBS”
2021/07/08 10:54:32 [error] 570#570: *169 upstream timed out (110: Connection timed out) while connecting to upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/fe81f75495324a109de3812a3c5a9abd/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/fe81f75495324a109de3812a3c5a9abd/poll?dlp=t”, host: “1o24bbs.com”, referrer: “求去哪家ICC的建议 - CS News - 1024 BBS”
2021/07/08 10:54:33 [error] 570#570: *169 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 100.19.76.187, server: _, request: “POST /message-bus/1dd105462c5b4b4c978e1702fa885e14/poll?dlp=t HTTP/2.0”, upstream: “http://127.0.0.1:3000/message-bus/1dd105462c5b4b4c978e1702fa885e14/poll?dlp=t”, host: “1o24bbs.com”, referrer: “谈谈自己的找工作和icc经历 - CS News - 1024 BBS”
2021/07/08 10:54:36 [error] 570#570: *13261 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 220.196.56.24, server: _, request: “GET /?374e828961d36d46bd99dc8e13b7f640 HTTP/1.1”, upstream: “http://127.0.0.1:3000/?374e828961d36d46bd99dc8e13b7f640”, host: “1o24bbs.com”, referrer: “https://gateway.zscalertwo.net:443/auT?origurl=https%3A%2F%2F1o24bbs.com%2F%3F374e828961d36d46bd99dc8e13b7f640&wexps=1&_ordtok=SWk3WVhmhHBPJ4rLFk4nDRnkZ3&wexps=1”
In production.log, I see
Processing by CategoriesController#index as HTML
Parameters: {“41528430213d6ae753e074e11328b67b”=>nil}
Job exception: FATAL: the database system is shutting down
@pfaffman How do I flush redis? Thanks!
Have you done PostgreSQL 13 update.
Thanks!
My website is running the latest version of discourse. I have been running the latest version for at least one month.
I didn’t manually upgrade PogreSQL but I think Postgresql is version 13.
How do I make sure of that?
$ psql -V
psql (PostgreSQL) 13.3 (Debian 13.3-1.pgdg100+1)
BTW, I am running a single container app
Do you have any non-standard plugins?
Plugins I have:
- docker_manager
- discourse-adplugin
Do you think the issue is with redis or postgresql?
After # ./launcher enter app when I run curl http://127.0.0.1:3000 it times out
1 Like
Output of discourse-doctor:
==================== DOCKER INFO ====================
DOCKER VERSION: Docker version 19.03.13, build 4484c46d9d
DOCKER PROCESSES (docker ps -a)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
70709beca56d local_discourse/app "/sbin/boot" 15 minutes ago Up 15 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp app
1828993cd957 discourse/base:2.0.20210528-1735 "/bin/bash -c 'cd /p…" 2 hours ago Exited (1) 2 hours ago agitated_curie
70709beca56d local_discourse/app "/sbin/boot" 15 minutes ago Up 15 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp app
Discourse container app is running
==================== PLUGINS ====================
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-adplugin.git
No non-official plugins detected.
See https://github.com/discourse/discourse/blob/master/lib/plugin/metadata.rb for the official list.
========================================
Sorry. I don’t have any ideas. Have you rebooted lately?
I had to bring up a new VM and use db backup to create a new container
But it seems to be happening again after that …
had the same problem with a company website foreign exchange send money online, tried to rebot the server and change the timeout to 1000 sec and then try it out.