Discourse stuck on "Currently Upgrading"


#1

Been having an odd problem with our Discourse instance recently.

It seems to be stuck on Currently Upgrading (even when there are no upgrades available):

There is no way to upgrade from within the web ui. When there are upgrades available, when I click on it the console window is just a blank screen. I need to do a ./launcher rebuild to upgrade, and even then it sometimes hangs on the stop app command, and I have to reboot the server to get the forum back.

There is nothing in the logs that I can see that pertains to this error.


(Sam Saffron) #2

Bring out the :hammer:

cd /var/discourse
git pull
./launcher rebuild app

#3

I’ve run those commands many times but the problem still remains (just tried it again and still stuck on “Currently Upgrading”).


(Sam Saffron) #4

I thought when you go to the currently upgrading page you can “cancel” it, does that work around the issue?


#5

Thank you, that solved it! I feel a bit silly now. :blush:


(Allen - Watchman Monitoring) #6

no need to feel silly… the idea that a rebuild should reset the Upgrading... state has come up before:

and


(Sam Saffron) #7

Really at some point we should set it up to auto reset on failure.


(Stefano Costa) #8

But when I click Reset upgrade a modal window says

WARNING: You should only reset upgrades that have failed and are not running.

This will NOT cancel currently running builds and should only be used as a last resort.

:confounded:


(Sam Saffron) #9

Sure, if at any point you feel even a bit worries I recommend

cd /var/discourse
./launcher rebuild app

(Allen - Watchman Monitoring) #10

@sam that didn’t clear the rebuilding flag in my experience here “Currently upgrading” even after a rebuild:


(Dražen Lučanin) #11

I also seem to have these endless “currently upgrading” problems on every upgrade.

Doing the recommended commands does solve it …

… but it seems kind of weird to have to do these manual commands on every single upgrade, especially given that I receive emails to upgrade fairly frequently. Is there possibly something wrong in my setup if the upgrade process can’t properly upgrade itself? I tried to follow the official instructions for everything I did so far.

The console window when I click the “currently upgrading” link shows nothing (the progress bar is on 0):

These are the processes running:


(Jeff Atwood) #12

Do you have 2gb of swap space on your install? If not you can expect to have problems.


(Dražen Lučanin) #13

Hm … I don’t have swap enabled, since I’m running the instance on a DigitalOcean VM that uses SSD storage. My VM does have 2 GB of RAM as required by the Digital Ocean Discourse droplet. The Discourse-authored Digital Ocean instructions and Digital Ocean themselves don’t recommend enabling swap:

Although swap is generally recommended for systems utilizing traditional spinning hard drives, using swap with SSDs can cause issues with hardware degradation over time. Due to this consideration, we do not recommend enabling swap on DigitalOcean or any other provider that utilizes SSD storage. Doing so can impact the reliability of the underlying hardware for you and your neighbors. Hence, we recommend a minimum of 2 GB of RAM to run Discourse on a DigitalOcean Droplet.

Do you think I should still enable some amount of swap despite the warning and hope it won’t be used during normal operation or just not enable it and bump the RAM instead?


(Jeff Atwood) #14

Haha yes that is complete political BS :wink: swap does not cause degradation, it prevents out of memory errors under memory strain, like the ones you are seeing.


(Andreas Motl) #15

We wanted to inform you that we just ran into this issue for the first time and “launcher rebuild app” didn’t help, so we followed the advice from @eviltrout by flushing Redis:

Hitting the “Reset Upgrade” button sounded too risky for us as it suggests anything would be rolled back, which we didn’t want to do because everything seemed to work flawlessly after the upgrade got applied successfully, as always.

At this place, we want to tell you that every time we hit the “Upgrade Discourse” button, we are again amazed by the shipping and production deployment quality of Discourse. Kudos to the whole team and keep up the spirit!


Upgrade failed due to unclean database shutdown II