Migrate_to_s3 fails because of logo images

installation
1

I’m moving an existing site from storing uploads locally to storing them in S3, following this guide. Everything is working fine for new uploads, I have confirmed that they are showing up in S3 by checking manually. However, when I attempt to run rake uploads:migrate_to_s3 I get the following output:

Please note that migrating to S3 is currently not reversible!
[CTRL+c] to cancel, [ENTER] to continue

Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Running the migration, this may take a while...
rake aborted!
FileStore::ToS3MigrationError: Some uploads could not be migrated to the new scheme. You need to fix this manually.
/var/www/discourse/lib/file_store/to_s3_migration.rb:162:in `migrate_to_s3'
/var/www/discourse/lib/file_store/to_s3_migration.rb:65:in `migrate'
/var/www/discourse/lib/tasks/uploads.rake:127:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:106:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:80:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rails_multisite-4.0.1/lib/rails_multisite/connection_management.rb:90:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:104:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:100:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

Looking in the file where the error originates, lib/file_store/to_s3_migration.rb, I see that it seems to be running a check against the uploads table for rows where url NOT LIKE '//%' AND url NOT LIKE '/%#{seeded_image_url}%'. In my case I assume seeded_image_url resolves to uploads/default/original/_X/.

I ran the same check against the database manually, and found the following:

image
image830×272 17.2 KB

The bottom two I’m not worried about, I’m guessing those are just fluke artifacts of the switch - if they have no url then presumably the site doesn’t know where to find them so they can be safely removed. I’m more concerned about the logos, though. These are the default logos, which we are actually still using, but they’re distributed with the codebase rather than uploaded.

Is it safe to add a third condition in to_s3_migration.rb, line 146, that also ignores uploads where the URL column starts with /images/? Or should I just delete those rows because those files are handled separately from uploads now, and that’s just an artifact from an older version of Discourse? Or have I misunderstood what the task is doing (not super familiar with Rails) and the problem is something else?

1 Like
2

Came back the next day and tried again, and the migrate task worked fine. I guess the logo images were never the problem, and it was just those two extraneous ones. Presumably those got cleaned up by some sort of periodic task, because when I look now they’re gone.

In hindsight, the upload task checks for bad upload url’s with:

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/%#{seeded_image_url}%'").exists?

Presumably it’s the by_users that causes it to ignore the logo images, since those don’t have a valid user_id.

3 Likes