Setting up backup and image uploads to Minio S3

s3
minio

(Rishabh Nambiar) #1

Introducing Minio for Discourse

miniologo

Minio is an object storage server released under Apache License v2.0.
It is compatible with Amazon S3 cloud storage service. It is best
suited for storing unstructured data such as photos, videos, log files,
backups and container/VM images.

Step 1: Configuring Minio

Installing Minio Server

This guide assumes that you have already created a server in your preferred region on a VPS provider of your choice. We usually recommend Digitalocean. Store the IP address of the server for future use. (eg. 123.01.234.12).

We will be using the Minio Docker installer for simplicity,
you can also use the Minio Quickstart Guide to install from source or binaries instead.

Run docker pull minio/minio to download the latest stable Minio Docker image.

Setup Authentication

Create a Access Key ID/Secret Access Key pair of your choice and store it safely. Here’s an example pair, please try to generate long, secure keys.

Access Key ID: EXAMPLEFODNN7EXAMPLE
Secret Access Key: EXAMPLEKEYFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Run the following command to start Minio Server using the keys you just created:
The command has been taken from the Minio Custom Access and Secret Keys section of the Minio Quickstart Guide.

docker run -p 9000:9000 --name minio1 \
  -e "MINIO_ACCESS_KEY=EXAMPLEFODNN7EXAMPLE" \
  -e "MINIO_SECRET_KEY=EXAMPLEKEYFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  -v /mnt/data:/data \
  -v /mnt/config:/root/.minio \
  minio/minio server /data

If you wish to not generate your own keys, Run this command instead:
Note: Minio will auto-generate a new key pair every time you start the container.

docker run -p 9000:9000 --name minio1 \
  -v /mnt/data:/data \
  -v /mnt/config:/root/.minio \
  minio/minio server /data

This is the output you should see after running the above command and starting the container:

911c6d0c7995: Pull complete 
9952c099c0a8: Pull complete 
20127dd4dd25: Pull complete 
Digest: sha256:e8c43f24a6edb16a655553249a000aca24e176837c358d9e3e244dfac8b9c30c
Status: Downloaded newer image for minio/minio:latest

Created minio configuration file successfully at /root/.minio
Endpoint:  http://172.17.0.2:9000  http://127.0.0.1:9000
AccessKey: EXAMPLEFODNN7EXAMPLE 
SecretKey: EXAMPLEKEYFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Browser Access:
   http://172.17.0.2:9000  http://127.0.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://172.17.0.2:9000 EXAMPLEFODNN7EXAMPLE EXAMPLEKEYFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

These values will be required for logging into the Minio Browser on your IP address and using Minio Client.

Step 2: Configuring Discourse for Minio

After setting up your Minio keys, the next step is to configure your Discourse instance. Make sure you’re logged in with an administrator account and go the Settings section in the admin panel.

Type in “s3” in the textbox on the top-left to display only the relevant settings:

You will need to:

  • Check the “enable s3 backups” checkbox if you want to activate manual or automated backups
    • Enter the desired Space name (bucket) in “s3 backup bucket” if enable s3 backups is checked
  • Check the “enable s3 uploads” checkbox if you want to allow images to be uploaded and served by Minio
    • Enter the desired Space name (bucket) in “s3 upload bucket” if enable s3 uploads is checked
  • Paste in both “Access Key ID” and “Secret Access Key” in their respective text fields
  • In s3 endpoint, paste in the IP address of your VPS eg. http://123.01.234.12
  • For s3 region, leave the field at it’s default value. This is because your Minio Server instance is already located in the geographical location of your VPS and this setting is ignored.
  • NOTE: Enable the s3 force path style checkbox for using Minio.

What your settings should look like for Minio:
(Admin -> Settings -> Type ‘s3’ in filter)

Note: You can enable only backups or only uploads or both.

Step 3: Perform a Test Backup

Visit /admin/backups on your Discourse instance when logged in as an Administrator.
Click the Backup button to perform a private backup of your site.

To check if your backup was uploaded correctly, visit the Minio Browser app in your web browser.
url: http://<YOUR_SERVER_IP>:9000/

You should see a login page where you can enter your Access Key ID and Secret Access Key. You should see your uploaded backup file in the Minio Browser. If not, please check your credentials and url.

This test backup would create a bucket with the name you entered in site_settings.

Step 4: Setting an upload Bucket Policy (skip if not using Minio for image uploads)

Till now, we have set up Minio for private files but if you want to use Minio for uploads, you have to make the files publicly accessible. You will need to use the Minio Client app for configuring your bucket policies using a CLI.

Install Minio Client from the Client Quickstart Guide on your local development machine on the OS of your choice. To test if mc (Minio Client) was installed, run ./mc --help or mc --help.

Run: mc config host add myminio http://<YOUR_SERVER_IP>:9000 EXAMPLEFODNN7EXAMPLE EXAMPLEKEYFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

  • This command creates a Minio client called myminio for your server instance and IP address.

Before we set a policy for your upload bucket, you must first create your upload bucket using the Minio Browser app or the mc client. Run mc mb myminio/<YOUR_UPLOAD_BUCKET_NAME>.

Now run: ./mc policy download myminio/<YOUR_UPLOAD_BUCKET_NAME> to set an upload policy for your complete bucket.
Expected output: Access permission for myminio/<YOUR_UPLOAD_BUCKET_NAME> is set to download.

Enjoy

That’s it. From now on, your images or backups will be uploaded to and served from Minio.


Todo

  • Configure SSL
  • Configure domain??

Setting up file and image uploads to S3
Extend S3 configuration for other S3 API compatible services
Minio instead of S3?
What does rake uploads:migrate_from_s3 exactly do?
S3 migrations from/to minio problems
System upload not using s3 cdn url
(hosna) #2

Can you also tell us how to migrate previous data from local to minio?


(Ashesh Sen) #3

Hi, I successfully followed all your steps. However, the image now doesnt show properly. here is what happens to a new image

if i click that broken icon, it shows in a modal window but it doesnt show in the post itself properly. the previous image was uploaded without using minio and the later was uploaded after using minio.


(Rishabh Nambiar) #4

Can you confirm if you can see the image file uploaded in the Minio browser?

This problem can happen when:

  1. The image has not been uploaded
    OR
  2. The download policy has not been set for your bucket

I’m not sure if there’s a robust way to do this as of now :sweat_smile:
Some people have tried using the rake tasks for this, I will check of that works :+1:


(Ashesh Sen) #5

hi,

thanks for the prompt reply. yes the image has been uploaded and i did set the download policy. but the problem persists. actually, it is acting very funny. it uploads sometimes and sometime it doesn’t upload.


(Rishabh Nambiar) #6

That’s odd, Can you try opening the image url in a browser window and check if the image shows up? Also try opening the url in an incognito window to see if the file is publicly accessible or not.

The Minio setup has a lot of moving parts, if this still doesn’t work, try DigitalOcean Spaces. Meanwhile, I’ll test this out again :+1:


(Ashesh Sen) #7

Thank you so much for your tutorial. At least, I got the backup alright. Strange it’s not uploading to minio anymore. I did the process again. Maybe, I got the installation of client part wrong somewhere. I’ll try to read the documentation and try again later this week.


(hosna) #8

Did you check this @rishabhn ?

Also I have some other questions.

1- Why in the title you say “image uploads” ? what about other uploads like mp3 and mp4 files?

2- What happens to discourse full update, after we set uploads to minio? Does it contain everything?

3- what if we want to migrate from self hosted minio to other s3 providers?

4- what if we want to separate backup and upload endpoints? cause it would be much more safe if we have the backup somewhere else.


(hosna) #9

I did this configuration for my discourse. but now some of previous uploads like mp3 files that were on my local, doesn’t show. I didn’t do a migration by uploads:migrate_to_s3. Is this what I need to do to make all previous uploads working?


(Roshan Udayanga) #10

How can we authenticate the users when image accessing, I mean if we don’t handle that anyone can see the images in that folder path


(hosna) #11

I get the following error when I want to download my backup.

Aws::S3::Errors::NotImplemented (A header you provided implies functionality that is not implemented)
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'`

(hosna) #12

After enabling uploads for minio, I didnt migrate previous uploads. I thought they would be read from local disk. However now after one day, I saw that some (perhaps all) mp3 files cant be read from local anymore. Is it intended to be like this?


(Gerhard Schlager) #13

That’s what happens when you don’t run rake uploads:migrate_to_s3. You really need to execute it otherwise all your upload will be broken.

Well, I guess Minio isn’t 100% compatible with S3. It probably doesn’t support generating pre-signed download URLs. You’ll need to download the backup manually from wherever you stored them.