Extend S3 configuration for other S3 API compatible services

Not yet.

We have hundreds of people using S3 currently, so we must be very careful here so we don’t break current setups.

7 Likes

Update: Minio now works perfectly for uploads/backups :tada:


I can refactor the absolute_base_url method in various ways but I’ll wait till I get a review from the team (because we have to be careful with code/logic changes).

Let me know if I can do anything to help :+1:

3 Likes

the PR is merged.

@rishabh can you confirm if it works with uploads?

I have a question. If we enable this feature, what would happen to previous uploads? Those that are on our own disk. Do they automatically transfer? On the other hand, if some day we might want to move back the uploads to our own server and turn off this feature, How can we migrate data back?

Yes, it works with uploads! :+1:

No, previous uploads would be not be affected. They don’t transfer automatically and only the future uploads will be uploaded to the new provider when you switch. I will check if it’s possible/feasible to migrate from one method to another.

1 Like

awesome, so how do we setup digitalocean spaces any tutorials out there?

I’ve almost finished writing the tutorial, it’ll be up on Meta very soon!

Edit: For everyone following this thread, here’s a link to the tutorial: :tada:

8 Likes

I am going to try to migrate my existing images on S3 to Digital Ocean Spaces. An obvious path is to use the rake tasks to move the images back to the DO droplet and then run a eventually-available rake task to move the images from the DO droplet to DO Spaces. Since my droplet doesn’t have enough space for all the S3 images, I’ve come up a potential idea that lets me bypass my droplet’s space constraint. Can anyone comment whether it would work, or share if you see obvious issues?

The idea:

  • Download all S3 images into my own hard drive, then upload them all into Digital Ocean spaces, maintaining all folder structure.
  • Use a SQL query to change the ‘url’ of any file in the ‘uploads’ table from the S3 domain to the DO Space domain

That seems to be a faster way to do things, since the migration rake tasks had been very slow for me in the past, like 1mb / second.

I can try with one file, but want to ask anyone sees any obvious issues before I try it.

Thanks!

3 Likes

You could create additional space with digital Ocean whatever they call the bigger space you can attach. That’d be much simpler.

1 Like

The only issue is the upsizing is permanent. And if I want to scale down the size afterwards, I have to migrate the forum to another droplet. I am not sure how involved that is.

No. You can add just extra space in a separate partition. It’s on the same menu where you’d add a droplet.

1 Like

Oh, yes. I forgot to say that my droplet is in a region where there is no block storage available. To do that I have to set up a new droplet and migrate the droplet over.

1 Like

Oh. Bummer. Sorry, I forgot that some regions don’t have block storage (as you might guess since I can’t even remember what they call it!). If I understand your situation, though, I’d be inclined to move to one that does have block storage.

4 Likes

https://www.digitalocean.com/community/tutorials/how-to-migrate-from-amazon-s3-to-digitalocean-spaces-with-rclone

7 Likes

can we use minio instead of digital ocean?

1 Like

Yes this has been tested @rishabh can you update details both here and at: Minio instead of S3?

3 Likes

I need one more day to finish it, it’s a bit longer than the s3 guide. I’ll update the details in both places :+1:

4 Likes

Can we also migrate previous data from our server to minio?

Migrate from s3 or from local?

migrate from local to minio

@hosna This should help you set up Minio uploads/backups:
https://meta.discourse.org/t/setting-up-backup-and-image-uploads-to-minio-s3/95123

Note: I’ll be adding a section for enabling HTTPS on a Minio Server very soon!

6 Likes