There is a hidden site setting to allow admins to include S3 uploads in the backups. To enable it, you’d set the SiteSetting at the Rails console like below
cd /var/discourse
./launcher enter app
rails c
SiteSetting.include_s3_uploads_in_backups=true
exit
exit
This change takes effect immediately. No further action is necessary.
Another way to accomplish this would be to add it to app.yml file in the env: section. To do that you would add it in the env: section (say, under DISCOURSE_HOSTNAME)
DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true
To have this change take effect, you need to run ./launcher rebuild app command to have that setting applied.
I’m certain you need to destroy and start to change the env variables applied to the container.
Of course, if they have done upgrades with docker manager, those will be lost when the container is destroyed, which is why the rebuild is the safest recommendation. Maybe it’s best to recommend the rebuild since it’s the most foolproof.
Ok. I didn’t know.
But, what’s the diff between ‘Local Uploads’ and “Files Stored in S3 ‘Uploads’” folder?
Also, though I haven’t changed the setting in web_only or in rails console as told above YET, but only chose the one setting told here.
But still when I ran the backup, just 10 min ago, it showed as if it downloaded thousands of files, (which I presume could only be on my aws s3 storage bucket, and ‘Download’ word itself means it is downloading from remote S3 bucket folder).
Also it is showing it Failed to download many, So many files. So I presumed if these files were on local cloud server, why would it miss to include those files in the backup? See this screenshot:
If you have configured Configure an S3 compatible object storage provider for uploads and moved your local uploads to S3, then your uploads are there. They aren’t backed up because it’s assumed that S3 is backed up and downloading them is expensive and usually unneccesary.
If you need to get them off of S3 (like if you’re hosted on discourse.org and are moving to some other hosting) then you need a backup that includes the files stored on S3.
Why do you want to download your files from S3?
Yes, it looks like it’s failing to download the files from S3. My guess is that it downloaded none of them, though it’s possbible that your disk is full, maybe (I’d expect a disk full error, but maybe not?).
Mine being a very very small site, paying a few dollars, which are increasing every month with small updates, every month to Aws S3 storage is proving a bit unpractical to me. I’m going to shift the ‘Uploads’ (and perhaps ‘backup’ folder location also) to some other provider like Google One Drive, iDrive, Hetzner etc cheap alternatives.
I now find that even Aws’s diff types of storage (from the one most active towards arhival type of storage), if I had chosen them wisely in the beginning, would cost me just half for same qty/number of files.
When I chose to store my media ‘uploads’ to aws S3 bucket years ago, at that time (as much as I’m sure) ALSO MOVED all the existing media uploaded by the users to S3.
Moreover, I also checked now in my /var/discourse/shared/web_only/uploads/default/optimized/1X folder, it has only 63 png files and /var/discourse/shared/web_only/uploads/default/original/1X has just 3. (Except 1x no other similar folders exist in the ‘uploads or default’ folder.
Also, I’ve not changed any option in the rails console or in the Yml file YET to include s3 uploads in my backups. So, why is it showing that it has downloaded so many ‘Media Uploads’ FROM S3 !!!
And, failed to download so many Uploaded files? Screenshot here.
Moreover, my uploads folder on S3 is around 3 gb (32k files). whereas the backup logs showed that it downloaded only around 3.2k (10% of total).
Very confusing.
Is there any OTHER rails command to be double sure if that option wasn’t toggled on years back, when I shifted to Aws S3?
