2.6.0b2 upgrade VERY slow

I think they do :smiley::smiley::smiley:

2 Likes

Wait for this PR to be merged, then follow the steps listed:

3 Likes

:smiley: :+1:

About 100Gb :sunglasses:

:+1: will do

2 Likes

So as I look at automating this upgrade, it occurs to me that knowing when to do this skip-post-deployment thing is rather complicated.

For single-container installs, unless there is some bizarre migration (this is the first I’m aware of in my 4 years), there is no advantage to doing it this way. Most upgrades will have few migrations, and likely none that is compute-intensive.

For two-container installs, it’s always a good idea to SKIP_POST_DEPLOYMENT_MIGRATIONS, as you can disable your site if the migrations break the running container, which removes some of the joy of the ability to bootstrap the new container while the old one continues to run. BUT even still, not every upgrade includes migrations that break things, so the two-reboot solution provided by this (very welcome) commit, is likely to result in more downtime (two reboot cycles) than just living on the edge.

My current thinking is that a better solution would be to have bootstrap always do the migrations with SKIP_POST_DEPLOYMENT_MIGRATIONS=1 and then do a migration on the first boot. I don’t have an elegant solution for just how that’d work, though. Is there a quick way to tell if migrations are pending? I guess the boot script could have some if migrations are pending, run a migration just after it cranks up unicorn? That’s pretty fiddly, though, and likely not worth it for most installations most of the time.

2 Likes

Definitely not worth doing most of the time, but if your software’s upgrades rarely (but sometimes) take 10x as long as normal, that’s a real problem for a lot of users.

After this I won’t run Discourse upgrades until at least a week post release. Of course you could argue that I’m a dumbass for not waiting in the first place, I brought it on myself, I should know better, and that is all completely correct! Still, making upgrade times more consistent and predictable would be a great service for all the other dumbasses out there.

2 Likes

Alternatively, upgrading via the UI through docker_manager handles this automatically.

https://github.com/discourse/docker_manager/blob/master/lib/docker_manager/upgrader.rb#L82

7 Likes

That’s awesome! I virtually never upgrade that way,so I’d never have thought of it. Thanks.

2 Likes

Sorry, to be clear, if we add this environmental variable to our app container:

Either:

SKIP_POST_DEPLOYMENT_MIGRATIONS: true

or

SKIP_POST_DEPLOYMENT_MIGRATIONS: 1

When we run launcher it will skip all DB migrations when we rebuild ?

Is this the correct understanding?

Not quite. See Introducing Post Deployment Migration

7 Likes

Hi, I’ve managed to update everything using this method (meaning via UI), but afterwards it starts making this deployment thing with migrate instructions (the one that had been skipped from the beginning) and so on and it breaks after a while, stating that the upgrade wasn’t succeeded. However the forum works as it should and the admin dashboard shows everything as updated.

My question is: do I have to do anything else? Specifically about this deployment thing, do I need to run it through command line (to avoid crashing)? Are there any specific commands for it and will it break the forum while it is running?

1 Like

Do you still have the relevant logs? You need to run the migrations otherwise search will be broken on the site.

./launcher enter app
bundle exec rails db:migrate
2 Likes

No logs, but it were the same queries mentioned before in this topic.

Before I start this process and leave it running, will this temporarily break the website (like rebuild) or is it “safe”? Sorry to insist on this specific issue, but this is a production website and a downtime of 10-15 minutes is acceptable, however a day or two is not and has to be carefully considered if no other options are available.

1 Like

If you’ve already upgraded to the latest version, running the migrations will not have any downtime on your site. By upgraded, I mean that the new version was deployed but the post deploy migration in this topic failed.

3 Likes

Thanks. I ended up going back to the updates via UI, tried once more and the post deployment database migrations actually went through. It took several hours (and a lot of CPU :sweat_smile:) but they got completed and everything is now fully updated! :star_struck:

6 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.