S3 migrations from/to minio problems

What I meant is that the URL for your upload records are incorrect as

url: "/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3"

should be

url: "/uploads/default/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3",

My suggestion for now is to restore from a recent backup and try the migrate_to_s3 script again. If it doesn’t work, avoid running the migrate_from_s3 script so that we can help you debug what is happening.

4 Likes

I put a new pic and a new video in a topic and did the migration process.

According to the logs, the rake task successfully migrated both. However, links aren’t updated in discourse so now they are broken.

`Migrating /uploads/default/original/1X/c42728dcfbe57731a145e0eea56644326090753c.mp4 --> //m2.res.zabanshenas.com/talktest/original/1X/c42728dcfbe57731a145e0eea56644326090753c.mp4 took 1.6314957910217345 seconds`

Is it safe to manually transfer everything inside upload folder to minio and then change all upload links from local to that of minio by a rake task?

Did you find a way to fix this issue? I also want to try minio.

Unfortunately no. My advice is that if you have already a live discourse forum, you shouldn’t migrate it to minio. Cause you will face many problems. I didn’t test it with a completely new instance. But I guess its ok.

1 Like

I’ve seen some recent commits related to s3 in discourse repository. Is there any chance problems reported here are solved by those comments? :thinking:

Not sure, try it out and let us know!

1 Like

This time, I’m getting the following error:

root@test-app:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@test-app:/var/www/discourse# SiteSetting.migrate_to_new_scheme = true
bash: SiteSetting.migrate_to_new_scheme: command not found

I wanted to manually move all of the files to minio then do the migration. Now with this error, it seems I am not able to do that too.

1 Like

Before running those commands you need to run rails c to enter rails console.

2 Likes

thank you very much :blush:

root@test-app:/var/www/discourse# rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []
[3] pry(main)> 

now what should I do? I should exit? how?

That seems like the most valid action to do :wink:

exit

1 Like

:sweat_smile:
ok but still I get the same error message

root@test-app:/var/www/discourse# rails c
SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
[1] pry(main)> SiteSetting.migrate_to_new_scheme = true
=> true
[2] pry(main)> Jobs::MigrateUploadScheme.new.execute(nil)
=> []
[3] pry(main)> exit
root@test-app:/var/www/discourse# rake uploads:migrate_to_s3
Migrating uploads to S3 for 'default'...
Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
root@test-app:/var/www/discourse# 

that’s beyond my understanding at this point so it’s probably not working yet?

Idk perhaps @rishabh, @tgxworld, or @sam, can help me. Cause as far as I see, the newest changes to s3 task are done by them :confused:

What happens if I manually migrate all of the files to minio, and then turn on s3 option without doing the migration rake task? Is it ok?"

I tried to test migrate to s3 for another time to test if this commit which was done by @rishabh, fixes some of the problems above, However, I got the following error

rake uploads:migrate_to_s3
rake aborted!
NoMethodError: undefined method `include?' for nil:NilClass
/var/www/discourse/lib/tasks/uploads.rake:217:in `migrate_to_s3'
/var/www/discourse/lib/tasks/uploads.rake:210:in `block in migrate_to_s3_all_sites'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:73:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:210:in `migrate_to_s3_all_sites'
/var/www/discourse/lib/tasks/uploads.rake:206:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_to_s3
(See full trace by running task with --trace)

https://github.com/discourse/discourse/blob/f181e9cc08a5d3c31598587957f8999b7e62ac50/lib/tasks/uploads.rake#L217

@zogstrip can you take a look at this? I get this message

Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)

again and again even after I apply those settings, and I think its because of this commit.

That error is now fixed in https://github.com/discourse/discourse/commit/60790eb006eeaf288740bbfc3cc6eaa602179c13


Unfortunately, the Minio setup is currently broken, there are plans in the next few weeks to pull the SiteSetting s3_force_path_style that was introduced to support the Minio style urls.

Minio now supports the standard url structure that we use for S3. After the Minio-specific site setting is removed, the migration tasks will be tested and hopefully will work :crossed_fingers:

The migrate_to_s3 task will not work properly for Minio urls till this work is done.

3 Likes

Yes. I noticed that. Thank you.

So I keep an eye on recent S3 related commit hopefully to see this problem would be fixed soon, before I run out of space in my discourse instance :sweat_smile:

2 Likes

I updated to the latest. however still migration to s3 doesn’t work. cc : @zogstrip

Some uploads were not migrated to the new scheme. Please run these commands in the rails console

SiteSetting.migrate_to_new_scheme = true
Jobs::MigrateUploadScheme.new.execute(nil)
2 Likes