S3 migrations from/to minio problems

Contribute Bug

AI-generated summary

The discussion revolves around issues with S3 migrations from/to Minio. hosna initially reported two problems: migration from local to Minio resulted in broken links, and migration from Minio to local only transferred images, not other file types.

rishabh confirmed that the issue was due to a bug and that the rake tasks were only modified for DigitalOcean Spaces and AWS S3. tgxworld later fixed the bug in the uploads:migrate_to_s3 rake task.

However, hosna still encountered issues with migration, including incorrect URLs and broken links. rishabh suggested checking the s3_endpoint setting in Site Settings, and tgxworld pointed out that the URL for upload records was incorrect.

Despite attempts to fix the issue, hosna reported that the problem persisted, and rishabh explained that the Minio setup is currently broken and that plans are in place to pull the s3_force_path_style setting to support Minio-style URLs.

zogstrip suggested running a command to identify uploads that need to be migrated and recommended deleting records that are no longer in use. hosna reported that there were many such records and asked for guidance on how to proceed.

The discussion concludes with hosna acknowledging that there is still an issue with the URLs and considering tgxworld’s suggestion to restore from a recent backup and try the migration script again.

1

I recently tried to configure my discourse to use minio according to this topic:

https://meta.discourse.org/t/setting-up-backup-and-image-uploads-to-minio-s3/95123

lets say my discourse domain is example.com and my minio endpoint is minio.example.com and my bucket name is test. I faced the following problems

1- Migration from local to minio.

I did this

cd /var/discourse
./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake

It seems all files are migrated to minio however all links in my forum are broken now. they are like this:

https://example.com//minio.example.com/test/uploads/the-rest-of-url...

instead of

https://minio.example.com/test/uploads/the-rest-of-url...

2- Migration from minio to local.
I did this:

cd /var/discourse
./launcher enter app
rake uploads:migrate_from_s3
rake posts:rebake

And all I can see is only images migrated from minio to local. not any zip or mp4 file.

4 Likes
Migrate_to_s3 for Digital Ocean Spaces woes
Migrate to minio
Backup is failing due to bad HTTPS
2

This is indeed a bug and I will look into it this week, the rake tasks were actually never modified for Minio so I’ll start with doing that. As of now, the rake tasks only work with DigitalOcean Spaces and AWS S3.

8 Likes
4

Any chance we get this bug fixed soon?

5

Hi, I’m working on it right now, is your Minio endpoint running with a domain name? like abc.com

6

This was due to a bug in the uploads:migrate_to_s3 rake task and has been fixed in

https://github.com/discourse/discourse/commit/918633aa12e869cad6d16fce5fae88fe2a26b82e

3 Likes
7

Yes. My minio endpoint is like m.example.com

Ok Im going to check it soon and will inform if its fixed for me. :blush:

Is this also fixed @tgxworld ?

1 Like
8

Still it has problem.

now when I do

cd /var/discourse
./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake

the mp4 video url is turned to

//minio.example.com/test/uploads/the-rest-of-url....mp4

instead of

https://minio.example.com/test/uploads/the-rest-of-url....mp4
9

Did you rebuild before running?

10

I upgraded with web interface. So I thought maybe no rebuild is needed.

11

I also did a rebuild but after it I get same result.

Before migrating to minio:

Screenshot_2018-10-10%20%D8%AA%D8%B3%D8%AA%20%D8%AF%DB%8C%D8%B3%DA%A9%D9%88%D8%B1%D8%B3
Screenshot_2018-10-10%20%D8%AA%D8%B3%D8%AA%20%D8%AF%DB%8C%D8%B3%DA%A9%D9%88%D8%B1%D8%B31361×468 76.5 KB

Link is correct and start with https:// and it shows the video in the post.

After migrating to minio and rebake all posts:

Screenshot_2018-10-10%20%D8%AA%D8%B3%D8%AA%20%D8%AF%DB%8C%D8%B3%DA%A9%D9%88%D8%B1%D8%B3(1)
Screenshot_2018-10-10%20%D8%AA%D8%B3%D8%AA%20%D8%AF%DB%8C%D8%B3%DA%A9%D9%88%D8%B1%D8%B3(1)1349×541 82.3 KB

link is correct however it doesn’t start with https:// and as a result, the video doesn’t appear in the post. we can only see the link.

Btw as I guessed correctly, migration from minio to local is still not fixed. :point_down:

12

I’m not sure what the issue is but have you entered your s3_endpoint as https://m.example.com or just m.example.com in Site Settings?
If this is the problem then we need to improve the help text for that field.

3 Likes
13

I entered it as https://m.example.com. Is it correct? or I should go for m.example.com ? :thinking:

14

Yes it is. @tgxworld if you are still looking into this, the one thing different about this Minio setup is that the force_path_style setting is true. I don’t see how this can make the https:// go away :thinking:

4 Likes
15

This bug is still not fixed. And It got even worse.

First, I thought maybe I just don’t need to migrate local data and discourse would read them from local disk even If I enable s3 uploads. However, after one day, I realized that my local files, except for images, won’t be loaded (mp3, mp4 etc).

So I decided to migrate the files anyway and then change all strings starting with my minio endpoint by adding https:// , as the problem I mentioned here :

However, migration reports something like
/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3 does not exist locally
for all files, Although I checked the path and the file exist. (at least this migration was correct two months ago!)

so I desperately disabled s3 and rebaked all post. However, discourse still doesn’t load local files. I still am sure that the file exists there. So now all of my upload links are broken.

@tgxworld did a major update recently to uploads.rake. So maybe those commits broke it.

16

Can you provide me with the output of

./launcher enter app
rails c
Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first
17 
rails c
Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first
[1] pry(main)> Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first

=> #<Upload:0x0000563a9e2ff510
 id: 248,
 user_id: 1,
 original_filename: "03 Dishab.mp3",
 filesize: 4674760,
 width: nil,
 height: nil,
 url: "/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3",
 created_at: Wed, 14 Nov 2018 21:47:40 UTC +00:00,
 updated_at: Wed, 14 Nov 2018 21:47:40 UTC +00:00,
 sha1: "dd758eae1e79c7a2b844d571da9c67ca1df02d39",
 origin: nil,
 retain_hours: nil,
 extension: "mp3",
 thumbnail_width: nil,
 thumbnail_height: nil>
[2] pry(main)>
18

Do you know if that is the only file name with a space in it? It seems if spaces in file names had the potential to cause a problem it would have come up before, but maybe not?

19

I dont think this is the problem. Cause as far as I saw, All files are broken now. Its really rare I would have space in each and every upload.

1 Like
20

Did you carry out any remaps manually? /original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3 is not a valid URL that we would generate.

4 Likes
21

I don’t know what is a remap. So I think I didn’t do it.

Why? can you explain more? :thinking:
In fact, all of the uploads up until now are in uploads/default/original/1X