502 Bad Gateway, suddenly

Installation
1

Hello there.

Yesterday my Discourse forum was working fine: it’s more than one year that runs successfully on a Digital Ocean machine and I never had problems.

Before going to sleep yesterday, I checked my forum and all was fine. I wake up this morning and I get a 502 Bad Gateway error. I didn’t attempt to upgrade, and not even some kind of operation on the VM.

I mean… how is this even possible? I am clueless.

I tried to do a ./launcher restart app but it didn’t help, so I ran a ./launcher rebuild app and I get this:

root@Forum:/var/discourse# ./launcher rebuild app
Ensuring launcher is up to date
Fetching origin
remote: Counting objects: 154, done.
remote: Total 154 (delta 67), reused 67 (delta 67), pack-reused 86
Receiving objects: 100% (154/154), 19.20 KiB | 0 bytes/s, done.
Resolving deltas: 100% (109/109), completed with 23 local objects.
From GitHub - discourse/discourse_docker: A Docker image for Discourse
12fa363…1f5042f master → origin/master

  • [new branch] configurable-pg-sleep → origin/configurable-pg-sleep
  • [new branch] launcher-env-vars → origin/launcher-env-vars
  • [new branch] pg_ctl → origin/pg_ctl
  • [new branch] user-args-for-last → origin/user-args-for-last
    Updating Launcher
    Updating 12fa363…1f5042f
    Fast-forward
    image/base/Dockerfile | 19 ++++++++±—
    image/base/VERSION | 2 ±
    image/base/install-imagemagick | 2 ±
    image/base/install-nginx | 2 ++
    image/build.rb | 2 ±
    image/discourse/Dockerfile | 4 ±-
    image/discourse_bench/Dockerfile | 4 ±-
    image/discourse_dev/Dockerfile | 12 +++++±-
    image/discourse_dev/ensure-database | 2 ++
    image/discourse_dev/postgres_dev.template.yml | 41 +++++++++++++++++++++++++++
    image/discourse_fast_switch/Dockerfile | 4 ±-
    image/discourse_test/Dockerfile | 4 ±-
    launcher | 8 ++±–
    templates/web.letsencrypt.ssl.template.yml | 5 +±-
    templates/web.template.yml | 2 ±
    15 files changed, 87 insertions(+), 26 deletions(-)
    create mode 100644 image/discourse_dev/postgres_dev.template.yml

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image ‘discourse/discourse:1.3.10’ locally
1.3.10: Pulling from discourse/discourse
ba66ef388b03: Pulling fs layer
82f56b9b0da9: Pulling fs layer
82f56b9b0da9: Verifying Checksum
82f56b9b0da9: Download complete
ba66ef388b03: Verifying Checksum
ba66ef388b03: Download complete
Pulling repository docker.io/discourse/discourse
Tag 1.3.10 not found in repository docker.io/discourse/discourse
Your Docker installation is not working correctly

See: Troubleshoot docker installation issues

It looks like this error is Docker-related but I wouldn’t know how to solve it. Any help?

1 Like
2

How much disk space, how much memory? Check memory and disk space. Disk space in particular gets used up over time…

4 Likes
3

You were right. Disk space was almost unexisting. I removed some old backups and now I have this:

System load: 0.89
Processes: 111
Usage of /: 92.2% of 29.40GB
Memory usage: 83%
Swap usage: 29%
=> / is using 92.2% of 29.40GB

The forum went online almost instantly after that. Thanks!

However, as you can see from the stats above, I’ll probably have the same problem in a while. What can I do to free disk space, apart from reducing the number of stored backups? I once had the download remote images to local feature activated, but not anymore (since a few months, actually). Maybe I could delete something from there?

4

clean up old docker images

cd /var/discourse
./launcher cleanup

and clean up old Ubuntu updates

apt-get autoclean
apt-get autoremove
2 Likes
5

Terminated successfully, but

Kept running for ~4h. At the end my ssh session with the VM terminated because of a network issue and I couldn’t see the output of the command (if there was one… for what I know, it might be still running).

However, from the Discourse dashboard I’m tracking the free space on the disk and it’s… odd.

It says I have 261 MB free but this morning after deleting the backups I had more than 1 GB.

The /var/discourse directory is 4.3GB only. Could it be that something like the OS is downloading upgrades that I must apply to clear up space?

6

That is very very odd. Can you try again? @sam any reason this command wouldn’t execute?

1 Like
7

Tried again some minutes ago and I got this

WARNING: We are about to start downloading the Discourse base image
This process may take anywhere between a few minutes to an hour, depending on your network speed

Please be patient

Unable to find image ‘discourse/discourse:1.3.10’ locally
Repository docker.io/discourse/discourse already being pulled by another client. Waiting.

Last time I ran the command, it tried to download the image and got stuck on a Pull Complete message for hours. I guess it is still running.

8

Reboot your server. And then try again.

4 Likes
9

Rebooting the server worked, the ./launcher cleanup command ran successfully! Thanks!

However, it only cleared 16 MB. :frowning: Is there any other way to clear disk space?

On a sidenote, when I start my SSH session with the VM I read:

115 packages can be updated.
64 updates are security updates.

New release ‘16.04.1 LTS’ available.
Run ‘do-release-upgrade’ to upgrade to it.

And the apt-get autoremove command returns:

Reading package lists… Done
Building dependency tree
Reading state information… Done
0 upgraded, 0 newly installed, 0 to remove and 114 not upgraded.

Is there any chance that performing the upgrades will free disk space?

10

I think that you need a bigger disk. One way to save space is saving less backups, like only last 3 or 5, but looks like your forum is big enough that you need to move to a bigger machine.

1 Like
11

This topic was automatically closed after 2698 days. New replies are no longer allowed.