S3 migrations from/to minio problems

Not sure, try it out and let us know!

1 Like

This time, I’m getting the following error:

root@test-app:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@test-app:/var/www/discourse# SiteSetting.migrate_to_new_scheme = true
bash: SiteSetting.migrate_to_new_scheme: command not found

I wanted to manually move all of the files to minio then do the migration. Now with this error, it seems I am not able to do that too.

Before running those commands you need to run rails c to enter rails console.

1 Like

thank you very much :blush:

root@test-app:/var/www/discourse# rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []
[3] pry(main)> 

now what should I do? I should exit? how?

That seems like the most valid action to do :wink:

exit

1 Like

:sweat_smile:
ok but still I get the same error message

root@test-app:/var/www/discourse# rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []
[3] pry(main)> exit
root@test-app:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@test-app:/var/www/discourse# 

that’s beyond my understanding at this point so it’s probably not working yet?

Idk perhaps @rishabhn, @tgxworld, or @sam, can help me. Cause as far as I see, the newest changes to s3 task are done by them :confused:

What happens if I manually migrate all of the files to minio, and then turn on s3 option without doing the migration rake task? Is it ok?"

I tried to test migrate to s3 for another time to test if this commit which was done by @rishabhn, fixes some of the problems above, However, I got the following error

rake uploads:migrate_to_s3
rake aborted!
NoMethodError: undefined method `include?' for nil:NilClass
/var/www/discourse/lib/tasks/uploads.rake:217:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:210:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:210:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:206:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

@zogstrip can you take a look at this? I get this message

Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

again and again even after I apply those settings, and I think its because of this commit.

That error is now fixed in FIX: Use GlobalSetting values instead of ENV variables in migrate_to_s3 · discourse/discourse@60790eb · GitHub


Unfortunately, the Minio setup is currently broken, there are plans in the next few weeks to pull the SiteSetting s3_force_path_style that was introduced to support the Minio style urls.

Minio now supports the standard url structure that we use for S3. After the Minio-specific site setting is removed, the migration tasks will be tested and hopefully will work :crossed_fingers:

The migrate_to_s3 task will not work properly for Minio urls till this work is done.

3 Likes

Yes. I noticed that. Thank you.

So I keep an eye on recent S3 related commit hopefully to see this problem would be fixed soon, before I run out of space in my discourse instance :sweat_smile:

2 Likes

I updated to the latest. however still migration to s3 doesn’t work. cc : @zogstrip

Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
2 Likes

What does this return?

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a

I got this:

rails c
Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a
[1] pry(main)> Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a
=> [#<Upload:0x0000562c3bbc4ce8
  id: 576,
  user_id: 250,
  original_filename: "IMG_۲۰۱۸۱۱۲۱_۱۸۱۶۱۴.jpg",
  filesize: 3928012,
  width: 4160,
  height: 2774,
  url: "/original/1X/1dd3317d1187a3876e566bfed3ab63f6cc7fa3e0.jpeg",
  created_at: Wed, 21 Nov 2018 14:51:05 UTC +00:00,
  updated_at: Wed, 21 Nov 2018 14:51:05 UTC +00:00,
  sha1: "1dd3317d1187a3876e566bfed3ab63f6cc7fa3e0",
  origin: nil,
  retain_hours: nil,
  extension: "jpeg",
  thumbnail_width: 690,
  thumbnail_height: 460,
  etag: nil>,
 #<Upload:0x0000562c3bbc4a18
  id: 541,
  user_id: 250,
  original_filename: "42830819_2684367061787501_565953382028611173_n.jpg",
  filesize: 51088,
  width: 640,
  height: 640,
  url: "/original/1X/5700585646fe5f67c06d9b1d31f66a934ef558e4.jpeg",
  created_at: Tue, 20 Nov 2018 16:01:15 UTC +00:00,
  updated_at: Tue, 20 Nov 2018 16:01:15 UTC +00:00,
  sha1: "5700585646fe5f67c06d9b1d31f66a934ef558e4",
  origin:
   "https://scontent-frt3-2.cdninstagram.com/vp/d666fc4425104d46b51e0f3a3e51042b/5BF6B743/t51.2885-15/e35/42830819_2684367061787501_565953382028611173_n.jpg",
  retain_hours: nil,
  extension: "jpeg",
  thumbnail_width: 500,
  thumbnail_height: 500,
  etag: nil>,
 #<Upload:0x0000562c3bbc4810

So you have 2 images that needs to be migrated. Are they still being used anywhere? If not, you can safely delete these records.

1 Like

you mean I should do this again?

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

and btw I should delete them manually? or there is an automatic way?

Sorry It seems Its more than that. I didn’t scroll down. They are too much

And this is what I get when I want to migrate uploads

[4] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[5] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []

and It seems there is no change in the result of

Upload.by_users.where("url NOT LIKE '//%' AND url NOT LIKE '/uploads/default/original/_X/%'").to_a

btw I think there is still something wrong with my urls as @tgxworld mentions here.