After a Discourse upgrade my Ubuntu instance is dead

Support
1

Using AWS EC2 Linux/Ubuntu instances successfully for years, I am facing an issue today, that I’m unable to resolve:

I did an upgrade of Discourse through the /admin/upgrade URL. This seemed to have run through successfully.

Unfortunately after that, the machine is dead.

No HTTP access, not SSH access.

I already tried to switch it on and off via AWS EC2 web GUI. No success.

It is currently impossible to connect via SSH to the machine, not via Putty and also not via the web based AWS EC2 instance connection terminal window.

I’m really clueless and already did wait for several hours.

The EC2 monitoring also shows no high server load to me:

image
image1348×1272 85.7 KB

When this happened in the past weeks/months (approximately 2 or 3 times in total), rebooting the machine through the EC2 GUI always did make the machine come alive again, but not this time.

My question

Any hints on how to make the machine accessible via SSH again?

(I know that this is most likely not a Discourse issue, but since it has occurred directly after a Discourse upgrade, maybe other users already had this very same behaviour and have some hints for me)

2

You might want to check your disk space utilization.

Many of the symptoms you describe occur when the filesystem is full.

HTH

1 Like
3

Thank you.

While this sounds reasonable, I currently see no way to check this, since I cannot connect to the machine at all.

4

You should be able to check this with your EC2 control / admin panel; buI I am not an AWS user and so cannot help further.

1 Like
5

After yet another reboot and some wait time, the system is back again, out of nowhere again.

To me, it seems that the disk has enough free space.

Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           394M  876K  393M   1% /run
/dev/xvda1       97G   31G   67G  31% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/loop1       18M   18M     0 100% /snap/amazon-ssm-agent/1566
/dev/loop0       98M   98M     0 100% /snap/core/10185
/dev/loop2       29M   29M     0 100% /snap/amazon-ssm-agent/2012
/dev/loop3       98M   98M     0 100% /snap/core/10126
overlay          97G   31G   67G  31% /var/lib/docker/overlay2/5a799ab040002ad2ddec94ae85bcbe987543651a0d9478ddc12ab12715da7340/merged
tmpfs           394M     0  394M   0% /run/user/1000
3 Likes
6

Great news @uwe_keim

Onwards and upwards!

2 Likes
7

For anyone interested, I’ve continued my question here:

Probably there will be some more details in the following hours/days over there.

1 Like