Error connecting to Redis

installation
1

When I try to start discourse, I can’t access it in my browser. I found some information in the log files which might cause this. Does somebody know what the problem is?

# /var/discourse/standalone/log/rails/unicorn.stderr.log
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 2 Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:362:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:343:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:139:in `block in call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:292:in `with_socket_timeout'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:138:in `call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/subscribe.rb:43:in `subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/subscribe.rb:12:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:3467:in `_subscription'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:2300:in `block in subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:2299:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.3/lib/message_bus/backends/redis.rb:287:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.3/lib/message_bus.rb:721:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.3/lib/message_bus.rb:669:in `block in new_subscriber_thread'"]
Failed to report error: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) 3 Job exception: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

# /shared/standalone/log/var-log/redis/current
40:M 11 Dec 2019 17:46:24.868 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
40:M 11 Dec 2019 17:46:24.868 # Server initialized
40:M 11 Dec 2019 17:46:24.868 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
40:M 11 Dec 2019 17:46:24.868 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
40:M 11 Dec 2019 17:46:24.885 * DB loaded from disk: 0.017 seconds
40:M 11 Dec 2019 17:46:24.891 * Ready to accept connections

If those very long log lines are too hard to read on here, I’ve also put them up on hasteb.in: hastebin

2

Something strange is going on here. I recommend upgrading docker, rebooting your server and rebuilding discourse.

3

Sucessfully upgraded Docker using apt upgrade docker (it wasn’t the newest version), rebooted Server via reboot, rebuilt using ./launcher rebuild app (got some redis warnings while rebuildings, but they are mostly the same as in the var-log/redis/current file), but I still can’t access my site. I tried accessing accessing it via apache2 as a Proxy (ProxyPass) which gives me a 503 Error and through the ip and the http port (I’ve disabled HTTPS) which just gives me an “ERR_CONNECTION_REFUSED” error from my browser.
This time the unicorn.stderr.log just outputs the following:

I, [2019-12-12T11:58:37.378657 #70]  INFO -- : Refreshing Gem list
I, [2019-12-12T11:58:56.751681 #70]  INFO -- : listening on addr=127.0.0.1:3000 fd=9
I, [2019-12-12T11:59:04.682642 #70]  INFO -- : master process ready
I, [2019-12-12T11:59:23.890076 #172]  INFO -- : worker=0 ready
I, [2019-12-12T11:59:24.852000 #179]  INFO -- : worker=1 ready

and the redis current file still outputs the warnings like before.

4

Does nobody have a solution for this? I still haven’t found a way to fix this.

5

It’s nothing obvious. What about /var/discourse/shared/standalone/logs/rails/production.log (if I can type)?

6

There is no logs Folder inside of shared.

7

It’s log instead of logs. You gonna have to help us to help you…

3 Likes
8

The production.log File is just empty.

9

And I still haven’t found a way to fix this yet. I’ve also been talking to some Forum Admins using Discourse and nobody I found seems to have had this issue.

10

I am also having this issue.
In: unicorn.stdout.log:

Starting up 1 supervised sidekiqs
Loading Sidekiq in process id 4160
2020-01-25T19:29:16.281Z pid=4160 tid=orb7qpo9c ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2020-01-25T19:29:16.288Z pid=4160 tid=orb7qpoj0 ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2020-01-25T19:29:16.287Z pid=4160 tid=orb7rr8k4 ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)
2020-01-25T19:29:16.283Z pid=4160 tid=orb7qpop4 ERROR: Error fetching job: Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL)

This is brand new Discourse install, standalone container with unix socket connection to outer nginx. I’m trying to embed Discourse in Ghost site for comments, the errors are correlated with displaying a post with a Discourse embed (and Discourse tries to queue a job to parse the referring page?) I had a similar config working for a year but that one blew up trying to upgrade, so I’m reconfiguring from (faulty) memory. Plenty of room for user error here …

Unicorn logs show redis connect error
production.log.txt (155.8 KB) unicorn.stderr.log.txt (14.5 KB)

redis itself seems happy (from standalone/log/var-log/redis/current) current.txt (37.0 KB)
and seems to be handling some connections on the expected port :6379. And some of the connections are from unicorn-worker, FWIW.

root@server3:/var/discourse/shared/standalone/log/var-log/redis# nsenter -t $(docker inspect -f '{{.State.Pid}}' app) -n netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      2250/redis-server * 
tcp        0      0 localhost.localdom:3000 0.0.0.0:*               LISTEN      2280/unicorn master 
tcp        0      0 0.0.0.0:postgresql      0.0.0.0:*               LISTEN      2253/postmaster     
tcp        0      0 localhost.localdo:45404 localhost.localdom:6379 ESTABLISHED 3004/unicorn worker 
tcp        0      0 localhost.localdo:45448 localhost.localdom:6379 ESTABLISHED 2981/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45408 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45346 localhost.localdom:6379 ESTABLISHED 2280/unicorn master 
tcp        0      0 localhost.localdo:45456 localhost.localdom:6379 ESTABLISHED 3013/unicorn worker 
tcp        0      0 localhost.localdom:3000 localhost.localdo:49628 TIME_WAIT   -                   
tcp        0      0 localhost.localdo:45426 localhost.localdom:6379 ESTABLISHED 3022/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45346 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45336 localhost.localdom:6379 ESTABLISHED 2280/unicorn master 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45392 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45458 localhost.localdom:6379 ESTABLISHED 3022/unicorn worker 
tcp        0      0 localhost.localdo:45452 localhost.localdom:6379 ESTABLISHED 3004/unicorn worker 
tcp        0      0 localhost.localdo:45380 localhost.localdom:6379 ESTABLISHED 2981/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45410 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45390 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45374 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45420 localhost.localdom:6379 ESTABLISHED 3013/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45400 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45390 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45332 localhost.localdom:6379 ESTABLISHED 2280/unicorn master 
tcp        0      0 localhost.localdo:45392 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdom:3000 localhost.localdo:49626 TIME_WAIT   -                   
tcp        0      0 localhost.localdom:6379 localhost.localdo:45386 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45428 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45414 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45368 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45332 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45376 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45396 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdom:6379 localhost.localdo:45418 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45448 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45388 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45374 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45408 localhost.localdom:6379 ESTABLISHED 3004/unicorn worker 
tcp        0      0 localhost.localdo:45386 localhost.localdom:6379 ESTABLISHED 2981/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45398 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45398 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdom:6379 localhost.localdo:45430 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45394 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45458 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45456 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45420 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45396 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45394 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45368 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdom:6379 localhost.localdo:45426 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45376 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45428 localhost.localdom:6379 ESTABLISHED 3022/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45404 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45400 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45388 localhost.localdom:6379 ESTABLISHED 2970/sidekiq 6.0.4  
tcp        0      0 localhost.localdo:45414 localhost.localdom:6379 ESTABLISHED 3013/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45336 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45418 localhost.localdom:6379 ESTABLISHED 3013/unicorn worker 
tcp        0      0 localhost.localdo:45384 localhost.localdom:6379 ESTABLISHED 2981/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45384 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdo:45430 localhost.localdom:6379 ESTABLISHED 3022/unicorn worker 
tcp        0      0 localhost.localdo:45410 localhost.localdom:6379 ESTABLISHED 3004/unicorn worker 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45380 ESTABLISHED 2250/redis-server * 
tcp        0      0 localhost.localdom:6379 localhost.localdo:45452 ESTABLISHED 2250/redis-server * 
tcp6       0      0 [::]:6379               [::]:*                  LISTEN      2250/redis-server * 
tcp6       0      0 [::]:postgresql         [::]:*                  LISTEN      2253/postmaster     
udp        0      0 localhost.localdo:57631 localhost.localdo:57631 ESTABLISHED 2253/postmaster     
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     10681809 2253/postmaster      /var/run/postgresql/.s.PGSQL.5432
unix  3      [ ]         DGRAM                    10681045 2247/rsyslogd        /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     10682768 2257/nginx: master   /shared/nginx.http.sock
unix  3      [ ]         STREAM     CONNECTED     10682773 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682778 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682772 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682776 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682770 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682774 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682775 2257/nginx: master   
unix  2      [ ]         DGRAM                    10681046 2247/rsyslogd        
unix  3      [ ]         STREAM     CONNECTED     10682777 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682779 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10682771 2257/nginx: master   
unix  3      [ ]         STREAM     CONNECTED     10766487 2970/sidekiq 6.0.4   
unix  3      [ ]         STREAM     CONNECTED     10683830 2280/unicorn master  
unix  3      [ ]         STREAM     CONNECTED     10764869 27949/postgres: 10/  /var/run/postgresql/.s.PGSQL.5432
unix  3      [ ]         STREAM     CONNECTED     10684631 2750/postgres: 10/m  /var/run/postgresql/.s.PGSQL.5432
root@server3:/var/discourse/shared/standalone/log/var-log/redis#
11

I have the same problem.
I have the error EADDRNOTAVAIL on localhost:6379 multiple times in my log files. I will post them if asked.

12

Are those connection problems intermittent or are they all just when the container is starting up and redis starts after unicorn?

13

The error is scattered around the log file, but only appears once when rebuilding.

14

I have the same problem！！ :sob:

15

Hi guys,

I just updated via ssh.
went to /var/discourse and ./launcher rebuild app.

and now no pages are delivered.
seems not to connect to redis.

any ideas?

Thanks from Hamburg,
René

16

Check disk space? What is your evidence that you’re not connecting to redis?

1 Like
17

I’m running into the same error on an new Docker installation of Discourse. It worked fine, then I took it down to set up for Postgres 12 and to enable a static maintenance error page. Now, I can’t get Discourse to run again.

There seems to be an issue with Rails connecting to Redis. From production.log:

Error connecting to Redis on localhost:6379 (Errno::EADDRNOTAVAIL) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:363:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:344:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:106:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:307:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:105:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:382:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:231:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:320:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:230:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis/client.rb:125:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:915:in `block in get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `block in synchronize'", "/usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:51:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/redis-4.1.4/lib/redis.rb:914:in `get'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:360:in `process_global_backlog'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:271:in `block in global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus/backends/redis.rb:284:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:741:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-3.2.0/lib/message_bus.rb:689:in `block in new_subscriber_thread'"]
Creating scope :open. Overwriting existing method Poll.open.
Creating scope :open. Overwriting existing method Poll.open.
Creating scope :open. Overwriting existing method Poll.open.

production_errors.log is empty. Redis seems to start up normally. It’s worth noticing that the syslog contains these lines:

May 17 13:35:30 wobbly-wiki-forum-app rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
May 17 13:35:30 wobbly-wiki-forum-app rsyslogd: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
May 17 13:35:30 wobbly-wiki-forum-app rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="51" x-info="https://www.rsyslog.com"] start

I wonder if they could indicate an issue? FWIW there’s ample available memory and disk space on the server.

EDIT: Now resolved! I was running through an nginx reverse proxy, but had not followed the instructions to the letter. I had forgotten to add the template "templates/web.socketed.template.yml", and I also commented out the entire expose: section of app.yml. Now it works.

4 Likes
18

Has anyone else figured out a solution to this? I am encountering this error on a vanilla docker based installation on Ubuntu Server 20.04. I am not running anything though an Nginx reverse proxy. I just used the default docker install script.

2 Likes