502 Bad Gateway, suddenly

Product Support Self-hosting

AI-generated summary

A user experienced a sudden 502 Bad Gateway error on their Discourse forum, which was resolved after freeing up disk space by removing old backups. The issue was traced to nearly full disk usage (92.2%). Attempts to clean up using ./launcher cleanup and apt-get autoclean/autoremove were partially successful, but the cleanup command hung for hours, requiring a server reboot to complete. Despite these efforts, only 16 MB of space was freed, and the user still faces space constraints. The forum’s size suggests the need for a larger disk or better backup management. Vereos codinghorror Falco

1

Hello there.

Yesterday my Discourse forum was working fine: it’s more than one year that runs successfully on a Digital Ocean machine and I never had problems.

Before going to sleep yesterday, I checked my forum and all was fine. I wake up this morning and I get a 502 Bad Gateway error. I didn’t attempt to upgrade, and not even some kind of operation on the VM.

I mean… how is this even possible? I am clueless.

I tried to do a ./launcher restart app but it didn’t help, so I ran a ./launcher rebuild app and I get this:

root@Forum:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
remote: Counting objects: 154, done.
remote: Total 154 (delta 67), reused 67 (delta 67), pack-reused 86
Receiving objects: 100% (154/154), 19.20 KiB | 0 bytes/s, done.
Resolving deltas: 100% (109/109), completed with 23 local objects.
From GitHub - discourse/discourse_docker: A Docker image for Discourse · GitHub
12fa363..1f5042f master → origin/master

  • [new branch] configurable-pg-sleep → origin/configurable-pg-sleep
  • [new branch] launcher-env-vars → origin/launcher-env-vars
  • [new branch] pg_ctl → origin/pg_ctl
  • [new branch] user-args-for-last → origin/user-args-for-last
    Updating Launcher
    Updating 12fa363..1f5042f
    Fast-forward
    image/base/Dockerfile | 19 ++++++++±—
    image/base/VERSION | 2 ±
    image/base/install-imagemagick | 2 ±
    image/base/install-nginx | 2 ++
    image/build.rb | 2 ±
    image/discourse/Dockerfile | 4 ±-
    image/discourse_bench/Dockerfile | 4 ±-
    image/discourse_dev/Dockerfile | 12 +++++±-
    image/discourse_dev/ensure-database | 2 ++
    image/discourse_dev/postgres_dev.template.yml | 41 +++++++++++++++++++++++++++
    image/discourse_fast_switch/Dockerfile | 4 ±-
    image/discourse_test/Dockerfile | 4 ±-
    launcher | 8 ++±–
    templates/web.letsencrypt.ssl.template.yml | 5 +±-
    templates/web.template.yml | 2 ±
    15 files changed, 87 insertions(+), 26 deletions(-)
    create mode 100644 image/discourse_dev/postgres_dev.template.yml

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image ‘discourse/discourse:1.3.10’ locally
1.3.10: Pulling from discourse/discourse
ba66ef388b03: Pulling fs layer
82f56b9b0da9: Pulling fs layer
82f56b9b0da9: Verifying Checksum
82f56b9b0da9: Download complete
ba66ef388b03: Verifying Checksum
ba66ef388b03: Download complete
Pulling repository docker.io/discourse/discourse
Tag 1.3.10 not found in repository docker.io/discourse/discourse
Your Docker installation is not working correctly

See: Troubleshoot docker installation issues

It looks like this error is Docker-related but I wouldn’t know how to solve it. Any help?

2

How much disk space, how much memory? Check memory and disk space. Disk space in particular gets used up over time..

3

You were right. Disk space was almost unexisting. I removed some old backups and now I have this:

System load: 0.89
Processes: 111
Usage of /: 92.2% of 29.40GB
Memory usage: 83%
Swap usage: 29%
=> / is using 92.2% of 29.40GB

The forum went online almost instantly after that. Thanks!

However, as you can see from the stats above, I’ll probably have the same problem in a while. What can I do to free disk space, apart from reducing the number of stored backups? I once had the download remote images to local feature activated, but not anymore (since a few months, actually). Maybe I could delete something from there?

4

clean up old docker images

cd /var/discourse
./launcher cleanup

and clean up old Ubuntu updates

apt-get autoclean
apt-get autoremove
5

Terminated successfully, but

Kept running for ~4h. At the end my ssh session with the VM terminated because of a network issue and I couldn’t see the output of the command (if there was one… for what I know, it might be still running).

However, from the Discourse dashboard I’m tracking the free space on the disk and it’s… odd.

It says I have 261 MB free but this morning after deleting the backups I had more than 1 GB.

The /var/discourse directory is 4.3GB only. Could it be that something like the OS is downloading upgrades that I must apply to clear up space?

6

That is very very odd. Can you try again? @sam any reason this command wouldn’t execute?

7

Tried again some minutes ago and I got this

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image ‘discourse/discourse:1.3.10’ locally
Repository docker.io/discourse/discourse already being pulled by another client. Waiting.

Last time I ran the command, it tried to download the image and got stuck on a Pull Complete message for hours. I guess it is still running.

8

Reboot your server. And then try again.

9

Rebooting the server worked, the ./launcher cleanup command ran successfully! Thanks!

However, it only cleared 16 MB. :frowning: Is there any other way to clear disk space?

On a sidenote, when I start my SSH session with the VM I read:

115 packages can be updated.
64 updates are security updates.

New release ‘16.04.1 LTS’ available.
Run ‘do-release-upgrade’ to upgrade to it.

And the apt-get autoremove command returns:

Reading package lists… Done
Building dependency tree
Reading state information… Done
0 upgraded, 0 newly installed, 0 to remove and 114 not upgraded.

Is there any chance that performing the upgrades will free disk space?

10

I think that you need a bigger disk. One way to save space is saving less backups, like only last 3 or 5, but looks like your forum is big enough that you need to move to a bigger machine.