Discourse stuck on "Currently Upgrading"

[Chili]

Been having an odd problem with our Discourse instance recently.

It seems to be stuck on Currently Upgrading (even when there are no upgrades available):

There is no way to upgrade from within the web ui. When there are upgrades available, when I click on it the console window is just a blank screen. I need to do a ./launcher rebuild to upgrade, and even then it sometimes hangs on the stop app command, and I have to reboot the server to get the forum back.

There is nothing in the logs that I can see that pertains to this error.

[sam (Sam Saffron)]

Bring out the

cd /var/discourse
git pull
./launcher rebuild app

[Chili]

I’ve run those commands many times but the problem still remains (just tried it again and still stuck on “Currently Upgrading”).

[sam (Sam Saffron)]

I thought when you go to the currently upgrading page you can “cancel” it, does that work around the issue?

[Chili]

Thank you, that solved it! I feel a bit silly now.

[watchmanmonitor (Allen - Watchman Monitoring)]

no need to feel silly… the idea that a rebuild should reset the Upgrading... state has come up before:

https://meta.discourse.org/t/upgrade-currently-upgrading-forever/35850/?u=watchmanmonitor

and

"Currently upgrading" even after a rebuild

Had you started an upgrade in the web side before you did it on the command line? I’m just trying to understand how the flag got stuck in that position. The easiest way to unlock it is to flush redis. $ ./launcher enter app $ redis-cli flushall

[sam (Sam Saffron)]

Really at some point we should set it up to auto reset on failure.

[steko (Stefano Costa)]

But when I click Reset upgrade a modal window says

WARNING: You should only reset upgrades that have failed and are not running.

This will NOT cancel currently running builds and should only be used as a last resort.

[sam (Sam Saffron)]

Sure, if at any point you feel even a bit worries I recommend

cd /var/discourse
./launcher rebuild app

[watchmanmonitor (Allen - Watchman Monitoring)]

@sam that didn’t clear the rebuilding flag in my experience here “Currently upgrading” even after a rebuild:

"Currently upgrading" even after a rebuild

We should have the rebuild clear the upgrading flag in Redis.

[metakermit (Dražen Lučanin)]

I also seem to have these endless “currently upgrading” problems on every upgrade.

Doing the recommended commands does solve it …

Bring out the

cd /var/discourse
git pull
./launcher rebuild app

… but it seems kind of weird to have to do these manual commands on every single upgrade, especially given that I receive emails to upgrade fairly frequently. Is there possibly something wrong in my setup if the upgrade process can’t properly upgrade itself? I tried to follow the official instructions for everything I did so far.

The console window when I click the “currently upgrading” link shows nothing (the progress bar is on 0):

These are the processes running:

[codinghorror (Jeff Atwood)]

Do you have 2gb of swap space on your install? If not you can expect to have problems.

[metakermit (Dražen Lučanin)]

Hm … I don’t have swap enabled, since I’m running the instance on a DigitalOcean VM that uses SSD storage. My VM does have 2 GB of RAM as required by the Digital Ocean Discourse droplet. The Discourse-authored Digital Ocean instructions and Digital Ocean themselves don’t recommend enabling swap:

Although swap is generally recommended for systems utilizing traditional spinning hard drives, using swap with SSDs can cause issues with hardware degradation over time. Due to this consideration, we do not recommend enabling swap on DigitalOcean or any other provider that utilizes SSD storage. Doing so can impact the reliability of the underlying hardware for you and your neighbors. Hence, we recommend a minimum of 2 GB of RAM to run Discourse on a DigitalOcean Droplet.

Do you think I should still enable some amount of swap despite the warning and hope it won’t be used during normal operation or just not enable it and bump the RAM instead?

[codinghorror (Jeff Atwood)]

Haha yes that is complete political BS swap does not cause degradation, it prevents out of memory errors under memory strain, like the ones you are seeing.

[amotl (Andreas Motl)]

We wanted to inform you that we just ran into this issue for the first time and “launcher rebuild app” didn’t help, so we followed the advice from @eviltrout by flushing Redis:

"Currently upgrading" even after a rebuild

Had you started an upgrade in the web side before you did it on the command line? I’m just trying to understand how the flag got stuck in that position. The easiest way to unlock it is to flush redis. $ ./launcher enter app $ redis-cli flushall

Hitting the “Reset Upgrade” button sounded too risky for us as it suggests anything would be rolled back, which we didn’t want to do because everything seemed to work flawlessly after the upgrade got applied successfully, as always.

At this place, we want to tell you that every time we hit the “Upgrade Discourse” button, we are again amazed by the shipping and production deployment quality of Discourse. Kudos to the whole team and keep up the spirit!

[Peter_Backgren (Peter Bäckgren)]

Just adding another use case here with all the known problems. Thankfully there was help here as always (eventually after asking but always ).

Using a DO 30G maybe 5 year old installation. Yes, I know that’s pretty much asking for trouble, I haven’t had time to even check out their last 2 year ago upgrade so…

Got the Discourse Currently Upgrading, could not reset it.

Cursed having to ssh in to fix things, and of course I had 24G used which did NOT free up enough using the usual methods.

Eventually found the autoclean and autoremove which gave me another 5G and then things got better.

Could those two perhaps be a question in the rebuild when you run out of disk space?

Anyway, I guess even docker upgraded as I had both upgrades waiting.

The only question here really is, why on earth did I even end up in some “auto upgrade”? I specifically upgrade with 3-6 month intervals depending on when I have time to sit here and figure out why things went wrong this time . Of the last four upgrades only one has gone through nicely (yes, did I mention disk space already lol).

End nag :

I simply want a running forum, one I can take care of from anywhere, without having to resort to ssh and public/private keys etc which I don’t have access to everywhere.

So a MS-style forced upgrade (even if there is a setting somewhere which I have NOT selected specifically) is a real no-no when the result possibly is a non-working forum when you get annoyed of the dashboard nags.

But never mind, I would not change this software to anything else. If nothing else it keeps my outdated Linux knowledge up to date a bit .

[meglio (Anton)]

After rebuilding via command line, the “Currently Upgrading…” is still there.

And the “Reset” button does nothing

[sam (Sam Saffron)]

You can work around probably by flushing redis

./launcher enter app
rails c
Discourse.redis.flushall

[pfaffman (Jay Pfaffman)]

FWIW I have a site that’s doing the same thing.