So, I have a Discourse backup from our initial server and we’re trying to move the system to a new system.
Unfortunately, there are two problems:
During restore, it says that 71 of 1073 uploads are not migrated to S3, and therefore hard-fails during the restore process.
Attempting to fix this on the main instance by rebaking, etc. the posts, it indicates some uploads are still not migrated to S3, even when I try and use the uploads:migrate_to_s3 rake mechanism.
Is there any way to get verbose information from migrate_to_s3 as to which uploads are not migrated, so I can manually fix it? It’s possible also that they are referring to a dead/failed S3 repository from where we were initially uploading to, except that things exploded at that point and we simply switched S3 mechanisms (to MinIO of course), and there was still old stuff sitting in the AWS S3 side of things. Which I don’t think I can easily migrate to our MinIO instance.
Alternatively, is there a way to disable the restore mechanism’s uploads on s3 checker because I just forced the migration of uploads myself already?
OK, so, figured it out after doing a deep dive in the DB. Looks like leftovers (71 uploads) from the original long-dead S3 environment (or rather, in this case, available but not primarily used S3 environment, because it was not going to be cost effective).
Ended up doing this:
From Origin Server:
./launcher enter app
sudo -u postgres psql discourse
SELECT url FROM uploads WHERE url NOT LIKE '%ExpectedS3DomainGoesHere%
(replace the ExpectedS3DomainGoesHere with the actual URL you use for your S3 config)
This will get URLs for you to work with, because we need to do some things.
Where the URLs are from older buckets on different URLs, use the Amazon S3 client (or the client for your S3 storage backend), and:
a. Sync the unexpected URLs’ buckets if available down to local storage.
b. Sync the items from Local to the new Bucket.
discourse remap OLD-URL-FROM-DB NEW-URL-FROM-DB
While it was suggested here to use DbHelper.remap, the remap function from discourse worked fine.
Make sure the data is migrated. rails uploads:migrate_to_s3
Rebake time! rails posts:rebake
Backup Site again on ‘origin’ machine/server. Download that latest update.
From New Destination Server:
Set up Discourse as expected, copy app.yml and such from the origin server over to the new server in /var/discourse/containers/ to make sure the rebuild hits the proper plugins, etc. needed.
Make sure that you comment out any DISCOURSE_BACKUP_LOCATION: s3 entries in the app.yml though, if you are working with local backup copies. Had some issues with S3 being weird with backup files being truncated so I took the local approach for a restore.
What often works in these kind of situations is temporarily disabling s3 uploads on the original instance before taking a backup. After restoring you can enable s3 again.