Upstream prematurely closed connection


(Marco) #1

Hello,

in the last few days I am experiencing connection problems to my droplet where I run a Discourse instance. I don’t know where to look at, but if I go in /var/discourse/shared/standalone/log/var-log/nginx and inpect the error logs I find several lines like:

2017/11/27 08:43:57 [error] 53#53: *19563 upstream prematurely closed connection
 while reading response header from upstream, client: 79.46.121.53, server: _, r
equest: "GET /user_avatar/www.rasando.it/giancarlo_49/45/583_1.png HTTP/1.1", up
stream: "http://127.0.0.1:3000/user_avatar/www.rasando.it/giancarlo_49/45/583_1.
png", host: "www.rasando.it"
2017/11/27 09:42:47 [error] 52#52: *20263 upstream prematurely closed connection
 while reading response header from upstream, client: 185.30.176.26, server: _, 
request: "GET /user_avatar/www.rasando.it/the_ripper/45/3827_1.png HTTP/1.1", up
stream: "http://127.0.0.1:3000/user_avatar/www.rasando.it/the_ripper/45/3827_1.p
ng", host: "www.rasando.it"
2017/11/27 09:42:57 [error] 53#53: *20273 upstream prematurely closed connection
 while reading response header from upstream, client: 185.30.176.27, server: _, 
request: "GET /user_avatar/www.rasando.it/bilos2511/45/2068_1.png HTTP/1.1", ups
tream: "http://127.0.0.1:3000/user_avatar/www.rasando.it/bilos2511/45/2068_1.png
", host: "www.rasando.it" ...

The only thing that I can do to recover is to reboot the machine. Please help!


(Jay Pfaffman) #2

How much ram do you have and what are the memory settings in app.yml?


(Marco) #3

I have 2 GB and I have 4 unicorn workers. It has always worked with this configuration though!


(Jay Pfaffman) #4

@sam I have a multisite instance that’s going down and another standalone install is also having strange memory problems. I haven’t yet checked to see if they have similar error messages, but this might be worth a look.


(Michael Brown) #5

These errors with avatars were due to a bug with @sam’s hijack helper work which I believe is now operating properly (try.discourse.org has stopped with the errors at least).

Please try pulling the latest code and using that.


(Marco) #6

I’m already up to date.


(Michael Brown) #7

In that case can you please share:

  • did anything change in the last few days before the errors started?
  • the exact Discourse version on which you’re running (git rev-parse HEAD from /var/www/discourse in the container)
  • any errors from Discourse itself (/var/discourse/shared/standalone/log/rails/production.log)

(Jay Pfaffman) #8

I was getting

Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:345:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:331:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:101:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:364:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:221:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:306:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:220:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:134:in `block in call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:280:in `with_socket_timeout'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:133:in `call_loop'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/subscribe.rb:43:in `subscription'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis/subscribe.rb:12:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:2775:in `_subscription'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:2143:in `block in subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'", "/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:2142:in `subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.8/lib/message_bus/backends/redis.rb:330:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.8/lib/message_bus.rb:516:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/message_bus-2.0.8/lib/message_bus.rb:464:in `block in new_subscriber_thread'"]

This site I just upgraded, so I don’t have the previous version at hand.

The other site I have that’s having trouble is a multisite instance. Recently I’ve added a new site, rebuilt the data container, botched memory settings, and all manner of stuff that makes it completely unhelpful with specific info. I’m upgrading it now in hopes that the other updates described in Wrong account avatar after upgrade somehow will fix things.

Edit: But while that container was bootstrapping, the database stopped responding. I restarted the data container and it’s working again.

Has there been some change to the database templates or versions recently? My problems with the mutlisite started a couple days ago, but I attributed them to some wacky memory settings, which I’ve subsequently fixed.


(Marco) #9

abbb2b28eb5f194ff396626fe0ce123393aa17c4

Today’s production log says:

Job exception: Connection timed out
Job exception: Error connecting to Redis on localhost:6379 (Redis::TimeoutError)
Job exception: Connection timed out


(Marco) #10

I still have this error, and I have to reboot daily.

2017/12/08 07:54:44 [error] 52#52: *28564 upstream prematurely closed connection
 while reading response header from upstream, client: 95.250.200.147, server: _,
 request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "www.rasan
do.it"

2017/12/08 08:01:46 [error] 51#51: *28600 upstream prematurely closed connection
 while reading response header from upstream, client: 195.77.54.109, server: _, 
request: "GET /user_avatar/www.rasando.it/barbastress/45/7047_1.png HTTP/1.1", u
pstream: "http://127.0.0.1:3000/user_avatar/www.rasando.it/barbastress/45/7047_1
.png", host: "www.rasando.it"

Could it be the avatar bug still there?


(Matt Palmer) #11

The Redis::TimeoutError problems are probably caused by a Docker bug. Restarting Docker itself may provide a certain amount of relief.


(Marco) #12

Thank you @mpalmer. I could write a cronjob to restart Docker every day, while they fix it.


(Jay Pfaffman) #13

Did you see this thread? Downgrade docker.