2.6.0b2 upgrade VERY slow

michaeld · August 22, 2020, 9:29am

I think they do

jomaxro · August 22, 2020, 10:34am

Wait for this PR to be merged, then follow the steps listed:

Paracelsus · August 22, 2020, 11:01am

About 100Gb

will do

pfaffman · August 22, 2020, 4:55pm

So as I look at automating this upgrade, it occurs to me that knowing when to do this skip-post-deployment thing is rather complicated.

For single-container installs, unless there is some bizarre migration (this is the first I’m aware of in my 4 years), there is no advantage to doing it this way. Most upgrades will have few migrations, and likely none that is compute-intensive.

For two-container installs, it’s always a good idea to SKIP_POST_DEPLOYMENT_MIGRATIONS, as you can disable your site if the migrations break the running container, which removes some of the joy of the ability to bootstrap the new container while the old one continues to run. BUT even still, not every upgrade includes migrations that break things, so the two-reboot solution provided by this (very welcome) commit, is likely to result in more downtime (two reboot cycles) than just living on the edge.

My current thinking is that a better solution would be to have bootstrap always do the migrations with SKIP_POST_DEPLOYMENT_MIGRATIONS=1 and then do a migration on the first boot. I don’t have an elegant solution for just how that’d work, though. Is there a quick way to tell if migrations are pending? I guess the boot script could have some if migrations are pending, run a migration just after it cranks up unicorn? That’s pretty fiddly, though, and likely not worth it for most installations most of the time.

Wingtip · August 22, 2020, 6:06pm

Definitely not worth doing most of the time, but if your software’s upgrades rarely (but sometimes) take 10x as long as normal, that’s a real problem for a lot of users.

After this I won’t run Discourse upgrades until at least a week post release. Of course you could argue that I’m a dumbass for not waiting in the first place, I brought it on myself, I should know better, and that is all completely correct! Still, making upgrade times more consistent and predictable would be a great service for all the other dumbasses out there.

tgxworld · August 24, 2020, 1:12am

Alternatively, upgrading via the UI through docker_manager handles this automatically.

github.com

discourse/docker_manager/blob/main/lib/docker_manager/upgrader.rb#L82


      
          # We automatically handle renames from `master` -> `main`
          if repo.upstream_branch == "origin/master" && repo.tracking_ref == "origin/main"
            log "Branch has changed to #{repo.tracking_ref}"
          
            # Just in case `main` exists locally but is not used. Perhaps it was fetched?
            if repo.has_local_main?
              run "cd #{repo.path} && git checkout main"
            else
              run "cd #{repo.path} && git branch -m master main"
            end
          
            run "cd #{repo.path} && git fetch origin --tags --force"
            run "cd #{repo.path} && git branch -u origin/main main"
            run("cd #{repo.path} && git reset --hard HEAD@{upstream}")
          else
            run("cd #{repo.path} && git fetch --tags --prune-tags --prune --force")
          
            if repo.detached_head?
              run("cd #{repo.path} && git reset --hard")
              run("cd #{repo.path} && git -c advice.detachedHead=false checkout #{repo.tracking_ref}")
            else

pfaffman · August 24, 2020, 3:23am

That’s awesome! I virtually never upgrade that way,so I’d never have thought of it. Thanks.

neounix · August 24, 2020, 4:07am

Sorry, to be clear, if we add this environmental variable to our app container:

Either:

SKIP_POST_DEPLOYMENT_MIGRATIONS: true

or

SKIP_POST_DEPLOYMENT_MIGRATIONS: 1

When we run launcher it will skip all DB migrations when we rebuild ?

Is this the correct understanding?

pfaffman · August 24, 2020, 4:28am

Not quite. See Introducing Post Deployment Migration

Paracelsus · September 2, 2020, 4:42pm

Hi, I’ve managed to update everything using this method (meaning via UI), but afterwards it starts making this deployment thing with migrate instructions (the one that had been skipped from the beginning) and so on and it breaks after a while, stating that the upgrade wasn’t succeeded. However the forum works as it should and the admin dashboard shows everything as updated.

My question is: do I have to do anything else? Specifically about this deployment thing, do I need to run it through command line (to avoid crashing)? Are there any specific commands for it and will it break the forum while it is running?

tgxworld · September 3, 2020, 1:08am

Do you still have the relevant logs? You need to run the migrations otherwise search will be broken on the site.

./launcher enter app
bundle exec rails db:migrate

Paracelsus · September 3, 2020, 8:54am

No logs, but it were the same queries mentioned before in this topic.

Before I start this process and leave it running, will this temporarily break the website (like rebuild) or is it “safe”? Sorry to insist on this specific issue, but this is a production website and a downtime of 10-15 minutes is acceptable, however a day or two is not and has to be carefully considered if no other options are available.

tgxworld · September 4, 2020, 1:18am

If you’ve already upgraded to the latest version, running the migrations will not have any downtime on your site. By upgraded, I mean that the new version was deployed but the post deploy migration in this topic failed.

Paracelsus · September 4, 2020, 6:48pm

Thanks. I ended up going back to the updates via UI, tried once more and the post deployment database migrations actually went through. It took several hours (and a lot of CPU ) but they got completed and everything is now fully updated!

jomaxro · October 4, 2020, 6:53pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Moving posts returns 502 bad gateway Installation performance , server-resources	70	4483	December 13, 2021
Self-hosted upgrade to 3.1.0.beta2 with typical multi-container install requires extra downtime Installation	9	1309	January 26, 2023
Upgrade failed due to unclean database shutdown Installation	21	4162	November 9, 2017
2.6.0 Beta4 update failed with what looks like db errors Installation	7	670	November 18, 2020
./launcher rebuild app for old 1.6 discourse not working Installation	12	733	January 25, 2023

2.6.0b2 upgrade VERY slow

Related topics