I searched, but can’t find a basic overview for how to move our uploads from DigitalOcean to S3. I successfully setup S3 for new uploads and our backups a few months back. I’d like to complete the move of our uploads (~1.4GB) over to S3. This was a conversion from SMF2 from the start. We now have two upload folders, one at the root in smf2 and one in /var/discourse. The SMF2 directory is 2.8GB. I suppose there might be two steps here? Do I need to do separate steps to move from the SMF2 directory and from the /var/discourse directory? I came across rake to s3, but can’t find a guide other than a bunch of people talking about errors they encountered and suggestions to correct them. Is there a guide?

Please read this article, it should guide you on how to set up the task [image] Configure an S3 compatible object storage provider for uploads Self-Hosting information_source This topic covers how to configure some common S3 compatible Object Storage providers (S3 clones). See Set up file and image uploads to S3 for more details about Amazon AWS S3 configuration, which is officially supported and used internally by Discourse for our hosting services. Provider Service Name Works with Discourse? Amazon AWS S3 Yes Digital Ocean Spaces Yes Linode Object Storage Yes Google Cloud Storage Yes Scaleway Object Storage Yes Vultr Obj… You should be checking the configuration part for hooks section and set up the bucket using env variables to avoid the issues you’ve already read about.

That’s great! Thank you! Only thing on there that I don’t know about is a CDN. I see we can use Amazon Cloudfront. I’m guessing that wasn’t setup when doing the previous S3 setup steps. I’ll look for a guide how to get that setup.

You can set up cloudfront by using your uploads bucket as origin and once it’s up you can set it as S3 cdn link. That’s all that’s there to do.

Thanks. Followed this video to do the Cloudfront setup: [How to Set up an Amazon CloudFront Distribution for Your Amazon S3 Origin] I’m running rake posts:rebake now. 84k posts, so it’s gonna take some time.

Uh oh… trying to start it again… root@discourse-app:/var/www/discourse# rake posts:rebake Rebaking post markdown for ‘default’ 10027 / 83358 ( 12.0%)/usr/local/bin/rake: line 2: 959 Killed RAILS_ENV=production sudo -H -E -u discourse bundle exec bin/rake “$@”

Just happened again… Any suggestions? root@discourse-app:/var/www/discourse# rake posts:rebake Rebaking post markdown for ‘default’ 12901 / 83359 ( 15.5%)/usr/local/bin/rake: line 2: 2569 Killed RAILS_ENV=production sudo -H -E -u discourse bundle exec bin/rake “$@” Some posts are now pulling content from Cloudfront, so at least that’s sort of working.

It looks like this is happening without me doing a rebake? Sidekiq is pounding away at pulling hotlinked images.

OK, got the site rebuilt and the S3 setup steps were followed. Everything should be loading from S3 now. So how do I verify that? Can I delete the old uploads directory on my droplet now to free up space?

I clearly did something wrong. I’m using more disk space now than I was before.

The upload to S3 should happen during the rebuild. You should be able to browse the website assets in the s3 buckets and all the site assets should load from the S3 or CDN link supplied. If it isn’t working, there is definitely something wrong with the way you configured it. Are you getting any errors?

SMF2 Conversion and Rake to S3 Help

Product Support Self-hosting

itsbhanusharma (Bhanu Sharma) May 14, 2020, 11:44am 15

Wow, this wasn’t helpful. You chose n for

can you generate a publicly available version? you can share link in dm, I may be able to verify if migration to s3 actually worked or failed for some reason?

Topic		Replies	Views
Site Blank After Rebuild Product Support	7	912	May 13, 2020
Migrating uploaded files from DO to S3 Product Support	2	2045	June 5, 2020
How to turn off S3 storage? Self-hosting	28	2961	August 12, 2020
Migrating uploads from S3 to local Self-hosting s3	32	2831	July 3, 2024
Move discourse uploads to free space on server Product Support	6	1212	February 16, 2022

SMF2 Conversion and Rake to S3 Help

Related topics