Enable hidden setting to include S3 uploads in the backups

Documentation Self-Hosting
,
1

:bookmark: This guide explains how to enable a hidden setting in Discourse to include Amazon S3 (Simple Storage Service) uploads in your backups.

Discourse has the ability to store media uploads on Amazon S3 for scalability and reliability. However, these uploads are not included in backups by default.

This guide covers enabling a hidden setting to include S3 uploads in backups, with options to configure it via the Rails console or the app.yml file.

Using Rails Console

To enable S3 uploads to be included in backups via the Rails console, you can follow these steps:

  1. Access your Discourse server via SSH.
  2. Enter the Discourse Docker container by running:
cd /var/discourse
./launcher enter app
  1. Launch the Rails console:
rails c
  1. Enable the setting by executing:
SiteSetting.include_s3_uploads_in_backups = true
  1. Exit the console and container by typing:
exit
exit

This change takes effect immediately. No further action is necessary.

Modifying the app.yml File

You can also make this change by adding modifying the app.yml file in the env: section.

  1. Access the Discourse app container directory:
cd /var/discourse
  1. Open the app.yml file located in containers:
nano containers/app.yml
  1. Under the env: section, add the line:
DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true
  1. Save the file and exit the editor.
  2. Apply the changes by rebuilding the app:
./launcher rebuild app

To have this change take effect, you need to run ./launcher rebuild app command to have the setting applied.

Last edited by @SaraDev 2024-10-23T21:54:54Z

Check documentPerform check on document:
7 Likes
2

Isn’t it sufficient to destroy and restart the container?

1 Like
3

I think you don’t need to destroy and the restart is enough. I will confirm it later.

Anyway, thanks @pfaffman for your other howto guide which I used as a template for this one.

2 Likes
4

I’m certain you need to destroy and start to change the env variables applied to the container.

Of course, if they have done upgrades with docker manager, those will be lost when the container is destroyed, which is why the rebuild is the safest recommendation. Maybe it’s best to recommend the rebuild since it’s the most foolproof.

3 Likes
9

Why, isn’t this setting is enough to include all the uploads in the backup:
image

10

That will include the local uploads but will not download files that are in s3 to include in the backup.

1 Like
11

Ok. I didn’t know.
But, what’s the diff between ‘Local Uploads’ and “Files Stored in S3 ‘Uploads’” folder?

Also, though I haven’t changed the setting in web_only or in rails console as told above YET, but only chose the one setting told here.

But still when I ran the backup, just 10 min ago, it showed as if it downloaded thousands of files, (which I presume could only be on my aws s3 storage bucket, and ‘Download’ word itself means it is downloading from remote S3 bucket folder).
Also it is showing it Failed to download many, So many files. So I presumed if these files were on local cloud server, why would it miss to include those files in the backup? See this screenshot:

image
image1091×605 39.8 KB

12

If you have configured Configure an S3 compatible object storage provider for uploads and moved your local uploads to S3, then your uploads are there. They aren’t backed up because it’s assumed that S3 is backed up and downloading them is expensive and usually unneccesary.

If you need to get them off of S3 (like if you’re hosted on discourse.org and are moving to some other hosting) then you need a backup that includes the files stored on S3.

Why do you want to download your files from S3?

Yes, it looks like it’s failing to download the files from S3. My guess is that it downloaded none of them, though it’s possbible that your disk is full, maybe (I’d expect a disk full error, but maybe not?).

1 Like
13

Thanks. Again.

Mine being a very very small site, paying a few dollars, which are increasing every month with small updates, every month to Aws S3 storage is proving a bit unpractical to me. I’m going to shift the ‘Uploads’ (and perhaps ‘backup’ folder location also) to some other provider like Google One Drive, iDrive, Hetzner etc cheap alternatives.
I now find that even Aws’s diff types of storage (from the one most active towards arhival type of storage), if I had chosen them wisely in the beginning, would cost me just half for same qty/number of files.

But I’d do that now.

1 Like
14

When I chose to store my media ‘uploads’ to aws S3 bucket years ago, at that time (as much as I’m sure) ALSO MOVED all the existing media uploaded by the users to S3.

Moreover, I also checked now in my /var/discourse/shared/web_only/uploads/default/optimized/1X folder, it has only 63 png files and /var/discourse/shared/web_only/uploads/default/original/1X has just 3. (Except 1x no other similar folders exist in the ‘uploads or default’ folder.

image
image1360×768 103 KB

Also, I’ve not changed any option in the rails console or in the Yml file YET to include s3 uploads in my backups. So, why is it showing that it has downloaded so many ‘Media Uploads’ FROM S3 !!!
And, failed to download so many Uploaded files? Screenshot here.

Moreover, my uploads folder on S3 is around 3 gb (32k files). whereas the backup logs showed that it downloaded only around 3.2k (10% of total).

Very confusing.

Is there any OTHER rails command to be double sure if that option wasn’t toggled on years back, when I shifted to Aws S3?

Any explanation will be helpful.