Discourse suddenly not accessible anymore

Installation
1

Hello!

My discourse instance is not working anymore (nothing has been done on my side).

In production.log, I have got this message:
Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed,

What I have tryied so far:

  • Server reboot
  • Update docker + server reboot + rebuild app

I am quite new with docker and discourse and so I do not see what else I can do :frowning:

Host OS: centos 7
Proxy: Apache
Docker: latest version

On the host, I have fail2ban installed and firewalld.
In the firewalld log, I got strange lines:

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -F DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:52 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t filter -X DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.

2020-07-23 14:00:53 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

2020-07-23 14:00:53 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

Any help or advice would be appreciate :slight_smile:
thx in advance!

2 Likes
2

It looks like some issue with iptables and docker. Those are the things to search. It’s not a Discourse issue.

2 Likes
3

I would guess that the redis issue is more linked to discourse than iptables / docker, no?

4

It suggests to me that docker networking is broken, so even if Discourse and redis are working properly, Discourse cannot access redis because networking and/or docker are broken.

My guess is that these problems appeared sometime after you installed firewalld. and that’s what broke your installation.

2 Likes
5

Thx Jay.

I stopped the firewalld daemon, restart docker and the faulty container and the error is still the same (in production.log):

Error connecting to Redis on localhost:6379 (Errno::ENETUNREACH) subscribe failed, reconnecting in 1 second.

I connected to the docker container:

docker exec --privileged -it <container id> bash

I can see redis running:

redis 45 0.2 0.1 45936 4352 ? Sl 05:29 0:02 /usr/bin/redis-server *:6379

Any idea?

6

I have done some progress. my config is using apache 2.4 as proxy. In my config, I have added the following:

RemoteIPHeader X-Forwarded-For

To keep track of the original IP address.
After httpd restart, I was able to access the discourse (with an alert about the ssl certificate expiration). I have fixed that one too and now everything is working well but with firewalld disabled.

I have found many posts about issues between docker and firewalld on centos 7. I will so switch back to iptables.

About the redis error message, after the above steps, this one only appears when docker is stopping. At restart, it’s not displayed so I would think this is ok.

Thx!

2 Likes
7

After reading this I stopped Docker on our Centos server

systemctl stop docker

then started it

systemctl start docker

problem solved.

Thanks @pfaffman

1 Like