502 error on nginx

Product Support Self-hosting

AI-generated summary

The discussion revolves around a 502 error on an nginx setup for a Discourse installation. elopio initially reports the issue, sharing the error log and netstat output. itsbhanusharma suggests that the error may be due to services starting up, which should resolve within 30 seconds. However, elopio confirms that the CPU load is well under 10%, ruling out extreme load as the cause.

Stephen asks about installed plugins and the server’s storage type, which elopio reveals to be a non-SSD drive with default plugins. neounix requests the yml file, which elopio shares. Stephen points out that Discourse requires SSD, but neounix disagrees, stating that a traditional hard drive should not cause a 502 error.

However, Stephen counters that slow IO can indeed cause timeouts and 502 errors. neounix suggests monitoring the Rails and nginx logs to debug the issue. elopio discovers permission errors in the unicorn log, resolves the issue by deleting the cache, and successfully accesses the Discourse installation.

Finally, elopio shares the successful migration of a Costa Rican hackerspace from Telegram to Discourse, thanking the community for their help.

1

Hello,

I’ve just run ./launcher rebuild app with no apparent errors. But when I try to open the site, I get a 502 error.

The nginx error log (shared/standalone/log/var-log/nginx/error.log) shows:

2020/08/12 05:47:43 [error] 653#653: *7 connect() failed (111: Connection refused) while connecting to upstream, client: [...], server: _, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:3000/", host: "[...]"

I’ve checked all the topics I found about discourse and nginx 502 errors but couldn’t find or understand anything that makes sense for my case.

This might be relevant, running from the container:

# netstat -plant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      644/nginx: master p 
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      644/nginx: master p 
tcp6       0      0 :::6379                 :::*                    LISTEN      -                   
tcp6       0      0 :::5432                 :::*                    LISTEN      -

Is there something that should be running in port 3000?

Could you please guide me on where to look for more information to debug this problem?

pura vida :slight_smile:

2

It shows 502 initially as the services inside the container are starting up. It should go away within 30 seconds. If it doesn’t it is possible that your server’s CPU is under extreme load and it is causing slowdown.

3

Thanks @itsbhanusharma.

I’ve run ./launcher restart app and monitored the load with top, and it’s well under 10%.

I think the CPU should be enough to handle the load:

$ cat /proc/cpuinfo  | grep 'name' | uniq
model name      : Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz
$ cat /proc/cpuinfo  | grep process | wc -l
4

Is there a better way to debug the start up of the container?
Any other ideas?

4

Which plugins are installed? Is this server on SSD?

5

I’ve just cloned the repository and run the setup commands, so I guess it just has the default plugins? Not sure where to check, but I’m sure I’ve added none.

The server is not SSD.

6

Hey @elopio

Maybe you can post your ‘yml‘ file without any sensitive info?

7

Sure, it’s here:

8

Discourse requires SSD, normal spinning drives don’t deliver sufficient IOPS.

9

And is that the cause for the 502 error? Or without an SSD should I see the website working, but very slow?

10

Using a traditional hard drive versus an SSD drive should not cause a 502 error. That’s not really plausible, as your question indicates @elopio.

Here is a little writeup which might be helpful:

The best thing to do, in my view, is to open up some terminals and run tail -f on your both your Rails and nginx log files, including the error logs and access logs; and then try to access and make sure when you see the 502 error you have your eyes on the tails of the log files.

Do you know where those logfiles reside and how to run tail -f commands on them in terminals?

Note, you asked earlier:

Rails run on port 3000 inside the Docker container and that port is not exposed outside the container. That is why you are not seeing port 3000 outside the container when you run netstat outside the container.

HTH

11

Speaking from experience slow IO absolutely can and does cause timeouts and a 502 error.

SSD-level IOPS are a firm requirement.

12

Wooo, looking at the rails logs I found that the unicorn log was huge, complaining about some permission errors. I deleted rm -rf tmp/cache/bootsnap-compile-cache/, and now I see the congratulations screen!!!

Thank you friends. I’ll play with this a little more before deciding to recreate it in an SSD server.

13

You are welcome @elopio

Glad you found the problem in the logs and you sorted this out. Well done.

Wishing you smooth sailing ahead!

14

Ok, this is working brilliantly. I want to show what we are doing:

https://bunqueer.jaquerespeis.org/

It’s the migration of the Costa Rican hackerspace from Telegram to Discourse :slight_smile: We still have to do sooo many things, but this time I’m sure we’ll be able to get rid of the chat for sure. Thanks a lot discourse people!