Set up BackBlaze S3 with BunnyCDN

[Bill]

It took me forever to figure out how to do all of this, so I wanted to make this guide to make it easier for people. Serving your images this way is supposed to make your website load faster. My Discourse setup combines a DigitalOcean droplet, BackBlaze B2 S3, BunnyCDN, and the free version of Cloudflare.

I went with these services after doing a bit of research into reliability, pricing, and benchmarks. They looked like the best options for me. Cloudflare helps with security, but will also slow the site down slightly from what I understand so you should look into that. Backblaze has partnerships with both cloudflare and bunny that allow for free data transfer.

B2/Cloudflare Partnership Offers the Lowest Cost Media Serving Platform

Although Cloudflare is called a CDN, it doesn’t work like regular CDN’s and it will probably not work alone for setting up S3 with Discourse. You will possibly need another CDN provider, BunnyCDN works well for me. If someone has successfully done it with just Cloudflare, let me know how

How to set uploads to S3:
First, you need to sign up at Backblaze, then make buckets, and insert the details into the Discourse settings:

Backblaze B2 Cloud Storage

Setting up backup and image uploads to Backblaze B2

So I was following the Digital Ocean Spaces guide, and I started looking around for cheaper providers for my (very) small community. I noticed this old post which confused me, because I had just looked at Backblaze B2 and saw that it was S3 compatible. Turns out they launched the feature 30 minutes before I had looked (what luck!). So I have decided to write a quick guide on how I got it to work with my current community: Step 1: Configuring Backblaze B2 Cloud Storage Create Account Go to B2 …

Then you sign up with BunnyCDN and follow this guide:

https://bunnycdn.com/?ref=qcp86164sq (referral link, I get a 20$ credit on my bill)

How to speed up your Backblaze B2 file delivery with BunnyCDN

After you create your BunnyCDN pull zone, you need to get the correct url to insert into your Discourse admin backend setting for “s3 cdn url”. Go to your bunnycdn dashboard>pull zones>manage pull zones>click your zone> copy the url next to “host name”

The url should like something like this: example.b-cdn.net but you will need to make sure you put https:// in front of it like this: https://example.b-cdn.net

Save the changes in Discourse admin setting area.

Press ctrl+f5 twice in your browser on your discourse site.

Do a test by uploading an image in the discourse text composer. Inspect the test image element, or view the images source to make sure it worked. The url should look different than your normal domain, it should be something like example.b-cdn.net / filename

Now, every time someone uploads an image, it will be in your backblaze bucket and not taking up room on your digitalocean droplet

Backup Uploads to S3
If you want to setup your Discourse backups to go into your backblaze bucket, you need to do this:

In the guide I posted higher in this thread, pay attention to how he explains you need two separate buckets. One bucket is public, this is your uploads bucket for your users to upload images on your forum. The other bucket is private, this is your backup’s bucket. It’s important to make automated backups of Discourse just in case anything breaks, then you can load your backup and fix it.

On your BunnyCDN dashboard, you should create a second pull zone. Your first pull zone was for uploads, this new pull zone will be for backups.

bunnycdn dashboard>pull zones>add pull zone

Follow the same guide I linked to earlier, BunnyCDN’s “How to Speed up your Backblaze B2 file delivery with BunnyCDN” but this time you would link the new pull zone to your backup bucket instead of your uploads bucket.

I recommend using the “high volume tier 5$/TB” on BunnyCDN’s pull zone creation page, I think backups don’t need the more expensive/faster tier “standard tier 10$/TB” that I used my for uploads bucket.

Set your backup settings in the discourse admin settings area. “backup location” should be set to S3 and “s3 endpoint” should point to what you have at backblaze, something like this:
s3.us-west-002.backblazeb2.com

CTRL+F5 twice on your website to clear cache

Start a backup, let it finish. Then wait 5 minutes and check your backblaze backups folder, you should see the new backup there.

Migrating Older Images:
This step is optional but recommended, especially if your low on storage on your server. So you got everything set up for new image uploads, but older images can be transfered to your buckets as well. To make this process work, you need to follow this:

@AntiMetaman

I have my backups working with S3 as well. So I have old and new uploads working with S3. I am using Cloudflare for SSL, DDoS, BunnyCDN for the upload and backup pull zones, and BackBlaze for the S3 storage. I’m all good now!

Note that it’s better to define the env variables in the app.yml as shown here: Using Object Storage for Uploads (S3 Clones)

Then rebuild so that the S3 configuration is globally defined and not just in admin panel. This way, when you migrate old files to s3, you can do it easier with 3 commands:

./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake

Setting up Cloudflare CDN
This step is optional since Backblaze has gotten a partnership that makes data transfer to Bunny free too, but for additional security and other features, you can make the data go through Cloudflare first, and then make Bunny pull the data from where Cloudflare drops it.

Using Backblaze B2 with the Cloudflare CDN

[AntiMetaman]

Just wanted to say I have done what Bill did, and confirm it works! The only catch is that it works for new files and not the old ones. The cool thing about Bill’s method is that it didn’t even need a rebuild. However, I hope that someone figures out how to get the old files to work with S3 too.

Backups don’t work either btw. At least not for me and I did the same exact thing as Bill.

[itsbhanusharma (Bhanu Sharma)]

You can migrate old files using the rake task.

[AntiMetaman]

If you mean this command, I already tried it and it didn’t upload the old assets:

sudo -E -u discourse bundle exec rake s3:upload_assets

I rebaked my posts too, but no dice.

[itsbhanusharma (Bhanu Sharma)]

The task is:

cd /var/discourse
./launcher enter app
rails c
rake uploads:migrate_to_s3

[Bill]

cd /var/discourse
./launcher enter app
rails c
rake uploads:migrate_from_s3

Does this work for you @AntiMetaman ?
I will add it to the guide if so.

I will try to figure out how to get backups to work correctly, and update the guide with that as well. I know for sure I need to make a second pull zone.

[itsbhanusharma (Bhanu Sharma)]

I have backups working fine with backblaze … What issue are you facing?

[Bill]

My backups are working fine with this method as well. You must have something set wrong. I will update the guide right now on how I set up backups. @AntiMetaman

[itsbhanusharma (Bhanu Sharma)]

@Bill

You must have something set wrong.

I’m sorry, I don’t follow this? How come I have something working fine when you are suggesting that I have something set wrong?

[AntiMetaman]

He’s talking to me @itsbhanusharma. My backups aren’t working. My configuration is the same as Bill’s. I didn’t use a separate pull zone for the backup bucket though so maybe that’s why. I just have the name of the backup bucket in the env variables.

I’ll try Bhanu’s suggestion to migrate the s3 assets later.

[AntiMetaman]

I created two pull zones, one for each of the buckets (upload and backup). I started a backup from my admin panel but it failed. The only thing that Discourse knows is the name of the backup bucket.

DISCOURSE_S3_CDN_URL: https://duelistsunite.b-cdn.net
DISCOURSE_S3_BUCKET: s3omega
DISCOURSE_S3_BACKUP_BUCKET: s3omegabackup
DISCOURSE_BACKUP_LOCATION: s3

Not sure what I’m missing here. Is there a S3_CDN_BACKUP_URL ? given that the url is different for the backup.

@itsbhanusharma Also, I tried your method, but migrate_from_S3 is undefined.

Also, I wasn’t sure if you meant migrate_to_s3 so I tried that too, but that was also undefined.

[itsbhanusharma (Bhanu Sharma)]

I’m sorry for the confusion, I meant rake uploads:migrate_to_s3

[AntiMetaman]

It aborts the rake it seems because it adds the amazonaws.com to the url when in fact, that is incorrect. This is the problem. If you use a S3 that’s not amazon, then the rake won’t work since it’s hardcoded to change the url.

[itsbhanusharma (Bhanu Sharma)]

Can you confirm/deny if you added settings to the app.yml env section or did you add the settings to admin > settings page?

[AntiMetaman]

I have my backups working with S3 as well. So I have old and new uploads working with S3. I am using Cloudflare for SSL, DDoS, BunnyCDN for the upload and backup pull zones, and BackBlaze for the S3 storage. I’m all good now!

Note that it’s better to define the env variables in the app.yml as shown here: Using Object Storage for Uploads (S3 Clones)

Then rebuild so that the S3 configuration is globally defined and not just in admin panel. This way, when you migrate old files to s3, you can do it easier with 3 commands:

./launcher enter app
rake uploads:migrate_to_s3
rake posts:rebake

[paulrudy]

Not sure what I’m missing here. Is there a S3_CDN_BACKUP_URL ? given that the url is different for the backup.

Did you resolve this question? I’m also not clear where to put the url for the pull zone that points to the backup bucket.

Edit: Am I correct in realizing that the CDN is only needed for the uploads bucket? The guide on this topic suggests a second CDN pull zone should be created for the backup bucket. If that’s wrong, perhaps the guide should be updated @Bill

[AntiMetaman]

Apparently it only needs the backup bucket name. From the name it can get the backup S3 url since it will be the same as the upload except for the name difference. This is why you don’t have to define a separate S3 backup url. This is of course assuming that both buckets are in the same S3 service.

[paulrudy]

But if the backup bucket is private, how would the CDN access it? I’m new to CDN’s and may be missing something, but I suspect that Discourse doesn’t use the CDN at all to back up.

[AntiMetaman]

They do. I went to my backup bucket and I see the backup uploaded there. It’s private but discourse can access it. You can setup permissions if the url is there. You can make it so only your site can access the bucket too or any https source.

[paulrudy]

Can you verify that the backup went through the CDN, vs directly from your Discourse install to BackBlaze B2?