Discourse suddenly not accessible anymore

Product Support Self-hosting

AI-generated summary

The user venizia03 reports that their Discourse instance is inaccessible due to a Redis connection error (Errno::ENETUNREACH) and iptables warnings related to Docker. Despite attempts to fix the issue through server reboots, Docker updates, and stopping firewalld, the problem persists. pfaffman suggests the issue may stem from Docker networking being broken, possibly due to firewalld. After further troubleshooting, venizia03 discovers that adding RemoteIPHeader X-Forwarded-For to the Apache configuration resolves the access issue, and the Redis error only appears during container shutdown, indicating it may be benign. The user concludes that switching from firewalld to iptables might be necessary. csmu confirms that restarting Docker resolved their similar issue, suggesting a potential fix for network-related problems.

1

Hello!

My discourse instance is not working anymore (nothing has been done on my side).

In production.log, I have got this message:
Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed,

What I have tryied so far:

  • Server reboot
  • Update docker + server reboot + rebuild app

I am quite new with docker and discourse and so I do not see what else I can do :frowning:

Host OS: centos 7
Proxy: Apache
Docker: latest version

On the host, I have fail2ban installed and firewalld.
In the firewalld log, I got strange lines:

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:53 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

2020-07-23 14:00:53 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

Any help or advice would be appreciate :slight_smile:
thx in advance!

2

It looks like some issue with iptables and docker. Those are the things to search. It’s not a Discourse issue.

3

I would guess that the redis issue is more linked to discourse than iptables / docker, no?

4

It suggests to me that docker networking is broken, so even if Discourse and redis are working properly, Discourse cannot access redis because networking and/or docker are broken.

My guess is that these problems appeared sometime after you installed firewalld. and that’s what broke your installation.

5

Thx Jay.

I stopped the firewalld daemon, restart docker and the faulty container and the error is still the same (in production.log):

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second.

I connected to the docker container:

docker exec --privileged -it <container id> bash

I can see redis running:

redis 45 0.2 0.1 45936 4352 ? Sl 05:29 0:02 /usr/bin/redis-server *:6379

Any idea?

6

I have done some progress. my config is using apache 2.4 as proxy. In my config, I have added the following:

RemoteIPHeader X-Forwarded-For

To keep track of the original IP address.
After httpd restart, I was able to access the discourse (with an alert about the ssl certificate expiration). I have fixed that one too and now everything is working well but with firewalld disabled.

I have found many posts about issues between docker and firewalld on centos 7. I will so switch back to iptables.

About the redis error message, after the above steps, this one only appears when docker is stopping. At restart, it’s not displayed so I would think this is ok.

Thx!

7

After reading this I stopped Docker on our Centos server

systemctl stop docker

then started it

systemctl start docker

problem solved.

Thanks @pfaffman