# S3 与 MinIO 之间迁移的问题
**URL:** https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333
**Category:** Bug
**Created:** [2018年九月17日 09:13 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333 "2018-09-17T09:13:48Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年九月17日 09:13 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/1 "2018-09-17T09:13:48Z")
I recently tried to configure my discourse to use minio according to this topic:
[https://meta.discourse.org/t/setting-up-backup-and-image-uploads-to-minio-s3/95123](https://meta.discourse.org/t/setting-up-backup-and-image-uploads-to-minio-s3/95123)
lets say my discourse domain is `example.com` and my minio endpoint is `minio.example.com` and my bucket name is `test`. I faced the following problems
**1- Migration from local to minio.**
I did this
```plaintext
cd /var/discourse
./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake
```
It seems all files are migrated to minio however all links in my forum are broken now. they are like this:
```
https://example.com//minio.example.com/test/uploads/the-rest-of-url...
```
instead of
```
https://minio.example.com/test/uploads/the-rest-of-url...
```
**2- Migration from minio to local.**
I did this:
```plaintext
cd /var/discourse
./launcher enter app
rake uploads:migrate_from_s3
rake posts:rebake
```
And all I can see is only images migrated from minio to local. not any zip or mp4 file.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [2018年九月17日 09:21 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/2 "2018-09-17T09:21:49Z")
This is indeed a bug and I will look into it this week, the rake tasks were actually never modified for Minio so I’ll start with doing that. As of now, the rake tasks only work with DigitalOcean Spaces and AWS S3.
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年九月24日 16:34 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/4 "2018-09-24T16:34:00Z")
Any chance we get this bug fixed soon?
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [2018年十月10日 09:24 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/5 "2018-10-10T09:24:39Z")
Hi, I’m working on it right now, is your Minio endpoint running with a domain name? like `abc.com`
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [2018年十月10日 09:31 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/6 "2018-10-10T09:31:23Z")
This was due to a bug in the `uploads:migrate_to_s3` rake task and has been fixed in
[https://github.com/discourse/discourse/commit/918633aa12e869cad6d16fce5fae88fe2a26b82e](https://github.com/discourse/discourse/commit/918633aa12e869cad6d16fce5fae88fe2a26b82e)
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十月10日 09:38 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/7 "2018-10-10T09:38:44Z")
Yes. My minio endpoint is like `m.example.com`
> [@tgxworld](#):
>
> This was due to a bug in the `uploads:migrate_to_s3` rake task and has been fixed in
Ok Im going to check it soon and will inform if its fixed for me. 😊
> [@hosna](#):
>
> And all I can see is only images migrated from minio to local. not any zip or mp4 file.
Is this also fixed @tgxworld ?
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十月10日 10:23 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/8 "2018-10-10T10:23:58Z")
> [@tgxworld](#):
>
> This was due to a bug in the `uploads:migrate_to_s3` rake task and has been fixed in
Still it has problem.
now when I do
```plaintext
cd /var/discourse
./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake
```
the mp4 video url is turned to
```plaintext
//minio.example.com/test/uploads/the-rest-of-url....mp4
```
instead of
```plaintext
https://minio.example.com/test/uploads/the-rest-of-url....mp4
```
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [2018年十月10日 10:24 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/9 "2018-10-10T10:24:24Z")
Did you rebuild before running?
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十月10日 10:25 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/10 "2018-10-10T10:25:00Z")
I upgraded with web interface. So I thought maybe no rebuild is needed.
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十月10日 10:56 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/11 "2018-10-10T10:56:37Z")
> [@tgxworld](#):
>
> Did you rebuild before running?
I also did a rebuild but after it I get same result.
Before migrating to minio:
Link is correct and start with https:// and it shows the video in the post.
After migrating to minio and rebake all posts:
link is correct however it doesn’t start with https:// and as a result, the video doesn’t appear in the post. we can only see the link.
Btw as I guessed correctly, migration from minio to local is still not fixed. 👇
> [@hosna](#):
>
> **2- Migration from minio to local.**
> I did this:
>
> ```plaintext
> cd /var/discourse
> ./launcher enter app
> rake uploads:migrate_from_s3
> rake posts:rebake
>
> ```
>
> And all I can see is only images migrated from minio to local. not any zip or mp4 file.
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [2018年十月10日 11:18 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/12 "2018-10-10T11:18:15Z")
> [@hosna](#):
>
> link is correct however it doesn’t start with https:// and as a result, the video doesn’t appear in the post. we can only see the link.
I’m not sure what the issue is but have you entered your `s3_endpoint` as `https://m.example.com` or just `m.example.com` in Site Settings?
If this is the problem then we need to improve the help text for that field.
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十月10日 11:29 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/13 "2018-10-10T11:29:08Z")
I entered it as `https://m.example.com`. Is it correct? or I should go for `m.example.com` ? 🤔
---
### Author:  [@rishabh](https://meta.discourse.org/u/rishabh)
#### Post date: [2018年十月10日 11:53 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/14 "2018-10-10T11:53:06Z")
> [@hosna](#):
>
> Is it correct?
Yes it is. @tgxworld if you are still looking into this, the one thing different about this Minio setup is that the `force_path_style` setting is `true`. I don’t see how [this](https://github.com/discourse/discourse/blob/master/app/models/site_setting.rb#L159) can make the `https://` go away 🤔
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十一月14日 22:23 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/15 "2018-11-14T22:23:29Z")
This bug is still not fixed. And It got even worse.
First, I thought maybe I just don’t need to migrate local data and discourse would read them from local disk even If I enable s3 uploads. However, after one day, I realized that my local files, except for images, won’t be loaded (mp3, mp4 etc).
So I decided to migrate the files anyway and then change all strings starting with my minio endpoint by adding https:// , as the problem I mentioned here :
> [@hosna](#):
>
> Link is correct and starts with https:// and it shows the video in the post.
>
> After migrating to minio and rebake all posts:
However, migration reports something like
`/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3 does not exist locally `
for all files, Although I checked the path and the file exist. (at least this migration was correct two months ago!)
so I desperately disabled s3 and rebaked all post. However, discourse still doesn’t load local files. I still am sure that the file exists there. So now all of my upload links are broken.
@tgxworld did a major update recently to uploads.rake. So maybe those commits broke it.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [2018年十一月15日 00:21 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/16 "2018-11-15T00:21:38Z")
> [@hosna](#):
>
> /original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3
Can you provide me with the output of
```plaintext
./launcher enter app
rails c
Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first
```
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十一月15日 06:42 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/17 "2018-11-15T06:42:58Z")
```plaintext
rails c
Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first
[1] pry(main)> Upload.where("url LIKE '%dd758eae1e79c7a2b844d571da9c67ca1df02d39%'").first
=> #
[2] pry(main)>
```
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [2018年十一月15日 09:24 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/18 "2018-11-15T09:24:11Z")
Do you know if that is the only file name with a space in it? It seems if spaces in file names had the potential to cause a problem it would have come up before, but maybe not?
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十一月15日 10:07 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/19 "2018-11-15T10:07:47Z")
I dont think this is the problem. Cause as far as I saw, All files are broken now. Its really rare I would have space in each and every upload.
---
### Author:  [@tgxworld](https://meta.discourse.org/u/tgxworld)
#### Post date: [2018年十一月16日 01:23 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/20 "2018-11-16T01:23:16Z")
Did you carry out any remaps manually? `/original/1X/dd758eae1e79c7a2b844d571da9c67ca1df02d39.mp3` is not a valid URL that we would generate.
---
### Author:  [@hosna](https://meta.discourse.org/u/hosna)
#### Post date: [2018年十一月16日 07:09 UTC](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333/21 "2018-11-16T07:09:02Z")
> [@tgxworld](#):
>
> Did you carry out any remaps manually?
I don’t know what is a remap. So I think I didn’t do it.
> [@tgxworld](#):
>
> is not a valid URL that we would generate.
Why? can you explain more? 🤔
In fact, all of the uploads up until now are in `uploads/default/original/1X`
[下一頁](https://meta.discourse.org/t/s3-migrations-from-to-minio-problems/97333.md?page=2)