Hi all, We have run a self-hosted Discourse instance at https://discourse.bokeh.org for a number of years. Generally speaking, it has been rock-solid an almost no effort to maintain, and in particular, performing updates is usually always a complete non-event that completes perfectly without any issues. However today after an update to 2.7beta7 (that seemed to complete without issue), our site has completely imploded. It limped along for a bit with pages mis-rendered and JS console errors, but after attempting a rollback, the UI it became non-functional. Logging in to the droplet, I have also tried to no avail: Rebuild ./launcher rebuild app This has failed in several ways over several tries. Discourse Doctor ./discourse-doctor Restore ./launcher enter app discourse restore This failed Wipe I also tried doing an “wipe” and then restoring ./launcher stop app ./launcher destroy app rm -r /var/discourse/shared/standalone/ after this I was at least able to get a rebuild to succeed which led to a “fresh install” state, e.g. " Congratulations, you installed Discourse!" So now I have tried running discourse restore again but this has failed again EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db 'default'. /var/www/discourse/lib/file_store/to_s3_migration.rb:131:in `raise_or_log' /var/www/discourse/lib/file_store/to_s3_migration.rb:86:in `migration_successful?' /var/www/discourse/lib/file_store/to_s3_migration.rb:357:in `migrate_to_s3' /var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate' /var/www/discourse/lib/file_store/s3_store.rb:240:in `copy_from' /var/www/discourse/lib/backup_restore/uploads_restorer.rb:62:in `restore_uploads' /var/www/discourse/lib/backup_restore/uploads_restorer.rb:44:in `restore' /var/www/discourse/lib/backup_restore/restorer.rb:62:in `run' script/discourse:145:in `restore' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start' script/discourse:286:in `' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli/exec.rb:63:in `load' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli/exec.rb:63:in `kernel_load' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli/exec.rb:28:in `run' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli.rb:494:in `exec' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli.rb:30:in `dispatch' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/cli.rb:24:in `start' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/exe/bundle:49:in `block in ' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors' /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.2.7/exe/bundle:37:in `' /usr/local/bin/bundle:23:in `load' /usr/local/bin/bundle:23:in `' Trying to rollback... Rolling back... Cleaning stuff up... Dropping functions from the discourse_functions schema... Removing tmp '/var/www/discourse/tmp/restores/default/2021-04-23-235404' directory... Marking restore as finished... Notifying 'system' of the end of the restore... Finished! [FAILED] What’s odd is during the restore, the site seemed to be getting back to normal, with old content showing up. The the failure happened and now nothing shows up, accounts are gone, etc. I could really use any guidance or suggestions. here. We have daily backups going back a week (futher in glacier if need be). We deleted an unused Category a few days ago, could that be the cause of problems? I will try an older back up to see, but any pointers to an iron-clad restore process would be welcome.

An older backup did not help. During the restore, things look “fine-ish” [Screen Shot 2021-04-23 at 5.20.53 PM] Right up until the end, then immediately go to this: [Screen Shot 2021-04-23 at 5.21.03 PM] As an aside, does the backup not contain posts that were imported originally from a mailing list, but have been on the Discourse site for years?? We had 24k posts, not 5k posts prior to this.

Is there a way to rebuild the app at different discourse at an older version of discourse, e.g at 2.7beta6?

Alternatively, I guess there is literally jsut a single post causing some problem EXCEPTION: 1 posts are not remapped to new S3 upload URL. S3 migration failed for db ‘default’. It it possible to find this post and just delete or remove it?

@pfaffman Yes we do have images on s3

Hmm. Well maybe there is just one broken post? I guess you’d need to restore the database, fix it, then rebuild the backup file with the new database. But it’s really hard to tell.

Are there instructions somewhere for how to do that? How can I determine what the one broken post is? Edit: alternatively is there someone with expertise that can be contracted for services?

Do you have a fresh droplet backup or snapshot what you can restore?

Do you have a fresh droplet backup or snapshot what you can restore? Alas, no. DO backups are only weekly and I had set up daily Discourse backups to s3 keeping a weeks worth fresh and a year back in glacier. Given my previous positive experience with many Discourse upgrades, I had though that would be both sufficient and better (and I had done successful test restores before)

[תמונה]Rebuild app without updating discourse? In app.yml uncomment #version: tests-passed let’s say version: 94301854938a0b36dd64666fb7a7c8406544a781 which is the commit just before the beta bump

קריסה לאחר שדרוג ל-2.7beta7

תמיכה

pfaffman (Jay Pfaffman) 24 באפריל,‏ 2021,‏ 12:42pm 15

I can help. On Monday, or sooner for hazard pay.

You can Contact Us - Literate Computing our S3 the email address at the bottom of the page.

נושא		תגובות	צפיות
Help restoring - system hung at midnight Self-hosting	26	2968	13 ביוני,‏ 2022
Can't upload backup Product Support	33	3398	27 בדצמבר,‏ 2021
Errors on Exporting Data from Teams to Self Hosted Discourse on Digital Ocean Product Support	39	2416	12 במאי,‏ 2021
Stuck and lost updating forum, problems with PG migration Self-hosting	15	715	12 ביוני,‏ 2025
Restore process cancelled at migrating uploads to S3 step Product Support	48	4032	24 בינואר,‏ 2026

קריסה לאחר שדרוג ל-2.7beta7

נושאים קשורים