# Ubuntu 20.04 kernel update met docker veroorzaakt een crash op EC2 en Lightsail

**URL:** https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526
**Category:** Self-hosting
**Created:** [9 juni 2022 om 15:00 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526 "2022-06-09T15:00:33Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy)
#### Post date: [9 juni 2022 om 15:00 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/1 "2022-06-09T15:00:33Z")

</div>

I ran into this issue last night, when my Ubuntu 20.04 LTS automatically upgraded itself, it installed a new kernel and I lost control of the system, it would just crash a few minutes after booting. I tried it again today with a fresh Discourse install and as soon as I upgraded the system it started crashing again.

Just a note for folks, don’t update your linux kernels just yet, this is a known bug - see this for more details.

> **[Bug #1977919 “Docker container creation causes kernel oops on li...” : Bugs :...](https://bugs.launchpad.net/ubuntu/+source/linux-aws-5.13/+bug/1977919)**
>
> Running the attached script on the latest AWS AMI for Ubuntu 20.04, I get a kernel panic and hard reset of the node.
> 
> \[12.314552\] VFS: Close: file count is 0
> \[12.351090\] ------------\[cut here \]------------
> \[12.351093\] kernel BUG at...

---

<div class="post-metadata">

### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy)
#### Post date: [9 juni 2022 om 15:03 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/2 "2022-06-09T15:03:28Z")

</div>

The question being is there a way to start the system without having it start Discourse/docker? Running on AWS Lightsail. The only other option is to rebuild the whole system again which is PITA right now given the backup/restore issues I’m facing.

EDIT: This is what I found, hit or miss depending on how fast it comes up.

```plaintext
while true; do
  ssh <instance> "sudo systemctl disable docker.service; sudo systemctl disable containerd.service"
done

```

---

<div class="post-metadata">

### Author: ![pfaffman](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/pfaffman/32/120154_2.png) [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [9 juni 2022 om 16:01 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/3 "2022-06-09T16:01:00Z")

</div>

I had this happen on two EC2 instances as well. They went down at 5AM EDT for a reboot and never came back up.

---

<div class="post-metadata">

### Author: ![Falco](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/falco/32/179432_2.png) [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [9 juni 2022 om 17:18 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/5 "2022-06-09T17:18:29Z")

</div>

Per the link this affects people running canonical “cloud kernels” in ubuntu machines. They removed a patch that affects OverlayFS.

While Canonical rolls a fix people can try a different kernel version or using Debian / other distro as a workaround.

---

<div class="post-metadata">

### Author: ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy)
#### Post date: [9 juni 2022 om 17:28 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/6 "2022-06-09T17:28:11Z")

</div>

I was able to interrupt the cycle using a quick SSH about 15 seconds after it starts to disable the docker/container services. Downgraded the kernel to 5.4 and it seems to be working

> **[How to downgrade the kernel on Ubuntu 20.04 to the 5.4 (LTS) version](https://discourse.ubuntu.com/t/how-to-downgrade-the-kernel-on-ubuntu-20-04-to-the-5-4-lts-version/26459)**
>
> No matter what your reasons are, you may be in a situation where you need to downgrade your kernel to a specific version. In this guide, I will show you how to downgrade a recent Ubuntu Pro 20.04 AMI’s kernel to the original LTS version (5.4) ...

---

<div class="post-metadata">

### Author: ![spamless](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/spamless/32/255876_2.png) [@spamless](https://meta.discourse.org/u/spamless)
#### Post date: [13 juni 2022 om 21:08 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/7 "2022-06-13T21:08:12Z")

</div>

> [@RBoy](#):
>
> I was able to interrupt the cycle using a quick SSH about 15 seconds after it starts to disable the docker/container services. Downgraded the kernel to 5.4 and it seems to be working
> 
> [How to downgrade the kernel on Ubuntu 20.04 to the 5.4 (LTS) version - Tutorials - Ubuntu Community Hub](https://discourse.ubuntu.com/t/how-to-downgrade-the-kernel-on-ubuntu-20-04-to-the-5-4-lts-version/26459)

Yes, as I just posted in your other thread on restore troubles, that was essentially what I did as well when this bug crashed my server. Well, I booted the old kernel; didn’t have to disable docker or containers. And the current kernel is safe again. [Here is a link](https://meta.discourse.org/t/help-restoring-system-hung-at-midnight/229486/30) to what I said in your other thread. In a bit I’ll try to write up my permanent solution to keep this from happening again.

Nasty kernel bug, that was!

---

<div class="post-metadata">

### Author: ![spamless](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/spamless/32/255876_2.png) [@spamless](https://meta.discourse.org/u/spamless)
#### Post date: [14 juni 2022 om 01:43 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/8 "2022-06-14T01:43:50Z")

</div>

You simply can revert to the previous kernel and the machine is restored. Or update to the current, fixed kernel, which came out on Thursday.

---

<div class="post-metadata">

### Author: ![spamless](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/spamless/32/255876_2.png) [@spamless](https://meta.discourse.org/u/spamless)
#### Post date: [14 juni 2022 om 01:54 UTC](https://meta.discourse.org/t/ubuntu-20-04-kernel-update-with-docker-causing-a-crash-on-ec2-and-lightsail/229526/9 "2022-06-14T01:54:20Z")

</div>

I have written up a tutorial on how to avoid kernel oops! issues like this that crash your server or keep it from coming back up.

I put the tutorial on my Discourse site, since that seemed convenient to me. My site has nothing to do with tech, though. So I unlisted the topic but published it to HTML.

Enjoy.

[https://discourse.bluebottlefly.com/pub/hardening-your-server](https://discourse.bluebottlefly.com/pub/hardening-your-server)

@RBoy, maybe you in particular will find this useful.

/dr
