Configure automatic backups for Discourse

[Tris20 (Tristan)]

Thanks for the hint. That pushed me towards the command line option which we can schedule to do whenever:

Backup discourse from the command line

These are some simple instructions on how to backup and restore your Discourse site from the command line. Backing up Enter the container: ./launcher enter app Run the backup: discourse backup exit This will print out the final filename of your backup. Download the backup and container configuration (if needed) to your local machine: scp =USER=@=HOST=:/var/discourse/shared/standalone/backups/default/=FILENAME= . scp =USER=@=HOST=:/var/discourse/containers/app.yml . Restoring First…

[bksubhuti (Bhante Bhikkhu Subhuti)]

I got this to work, but it seems that the uploads checkbox was not really needed, nor do I understand the purpose. What is the purpose? The only thing I want is backups to s3 instead of local for my server. The server only has weekly automatic backups…

The Json also had problems… I was able to get it to work using another website reference. However, nobody could upload any images because I had the uploads checkbox checked (as described here)…Unchecking that box fixed the image upload problem for users and their profile pics.

What is the purpose of the images upload? I’m seriously hoping images are inside the s3 backups.
I had to do the instructions twice because I didn’t understand “uploads” and only made one bucket. Then I had to do it again with 2 buckets, and then I had to remove the checkbox for uploads. It might be good if there was a separate more simple topic for s3 backups … and only backups.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutLifecycleConfiguration",
                "s3:CreateBucket",
                "s3:PutBucketCORS"
            ],
            "Resource": [
                "arn:aws:s3:::classicaltheravadabucket",
                "arn:aws:s3:::classicaltheravadabucket/*",
                "arn:aws:s3:::classicaltheravadabackupbucket",
                "arn:aws:s3:::classicaltheravadabackupbucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

[pfaffman (Jay Pfaffman)]

@bksubhuti

It might be good if there was a separate more simple topic for s3 backups … and only backups.

Configure automatic backups for Discourse

So you’d like to automatically back up all your Discourse data every day? Configuring daily backups Go to the /admin settings, backup, and set backup_frequency to 1. [image] Now backup will be taken every day. Store backups on the local server (default) By default backups are stored on the local server disk. If you’re self-hosted you can access them at /var/discourse/shared/standalone/backups/default. Store backups on S3-compatible storage (using admin panel config) You can also stor…

Though I think that topic should be updated to recommend that the S3 configuration be moved to app.yml rather than the database, that way you can do a command line restore of the database with only the yml file and not have to configure it with a user and s3 confige before doing a restore.

[bksubhuti (Bhante Bhikkhu Subhuti)]

@Configure automatic backups for Discourse

WARNING Storing backups and regular uploads in the same bucket and folder is no longer supported and will not work. By default backups are stored on the local server disk. In order to store backups on Amazon S3, you’ll need to create a unique,

I’m not sure what you are talking about . My backups are working see picture
.
I use s3 because digital ocean backups are only weekly, and if the server crashes and gets deleted, it is not useful.
On the other hand. I’m hoping that restoring from s3 or downloaded s3 bucket will be fine.
I’m not uploading the images, and I am hoping that the s3 backups are being backed up including images (although very few).

[RGJ (Richard - Communiteq)]

@bksubhuti

s3 backups are being backed up including images

Generally: no.
It does not make much sense to backup images in an S3 bucket into another S3 bucket?

[bksubhuti (Bhante Bhikkhu Subhuti)]

can you be less ambiguous?
The instructions had 2 s3 buckets. I could not get that to work.
I have only one s3 bucket. Hopefully pictures are included in that backup… is that correct?

I would imagine that local backups work the same, Right?

Please answer in full sentences regarding my questions. The tutorial was very confusing as well.

[RGJ (Richard - Communiteq)]

What is ambiguous about “no” ? (And what’s not ambiguous about “backups being backed up” )

Let me try again.

If you have configured uploads to be on S3 then uploads are not included in your backup.

[bksubhuti (Bhante Bhikkhu Subhuti)]

Let’s use the term “pictures” instead of uploads even though it can be other media.
This way we don’t confuse the text content as an upload which I’m uploading to s3.

So the 62 MB, backup files on s3 as pictured and uploaded in this thread does not include pictures ?

So how do I make sure the backups have these?
Do the local backups have the pictures too?

When I configured the s3 for "uploads (of media) " which was ambiguous as well. Nobody could post pictures because they were rejected from s3…

Is there a way to have both local and s3 daily backups?
I could care less if 5 days of pictures were lost, we are mostly a text based group.
But I would care if 5 days of text were lost. Digital Ocean only does 7-day backups if you pay them.
So even though I can backup daily, if the droplet gets hacked or damaged, then we lose those backups… I’m starting to think there is not much added value in s3.

I wish there were simple backups similar to WordPress which lets me backup to my google or dropbox account.

[RGJ (Richard - Communiteq)]

@bksubhuti

Let’s use the term “pictures” instead of uploads even though it can be other media.
This way we don’t confuse the text content as an upload which I’m uploading to s3.

No, that is a bad idea, if you upload a text file as an attachment, it’s an upload as well, it will cause confusion. And text in a post is stored into the database. So I’m sticking to the term uploads.

If your uploads are on S3, they are not included in backups. In that case the backups only contain a copy of the database. It does not matter whether your backups are local or on S3.

If your uploads are not on S3, they are included in the backups. In that case the backups contain a copy of the database, and a copy of the uploads. It does not matter whether your backups are local or on S3.

@bksubhuti

So even though I can backup daily, if the droplet gets hacked or damaged, then we lose those backups

If you are storing something on S3, be it uploads or backups of the database, they will not get lost if your DO droplet gets hacked or damaged. So I don’t see your point.

Since your posts are about backups and not about file and image uploads, I’m moving these to another topic.

[Canapin (Coin-coin le Canapin)]

I’d like to automatically move my S3 Backups to Glacier but I’m confused by the steps linked in the first post, which doesn’t explain much, maybe because there’s outdated stuff.

Which options should be checked here?

[Canapin (Coin-coin le Canapin)]

May I ask again in case someone did these steps and knows about it?

Also, do you know what makes these fluctuations in S3 fees?

Plus, since the launch of the forum (September 2020), the size of the backups has increased by roughly 15%, but the S3 bills have doubled, from 2.50$ to 5$. Any idea why that much?

That’s why I’d like to use Glacier.

---

Edit: I’ve followed the steps described here and I’ll see how it goes.

[Canapin (Coin-coin le Canapin)]

@Canapin

Edit: I’ve followed the steps described here and I’ll see how it goes.

Well, it doesn’t go.

My lifecycle configuration:

My S3 bucket:

No backup is on Glacier.

So… Two questions for those who have been able to achieve this automated S3 to Glacier transition:

1. What could be wrong in my configuration?

2. The minimum storage duration charge in Glacier is 90 days. Does that mean that if I do 1 backup a day, I’ll be eventually charged for 90 backups in Glacier each month?
   If this is the case, then this Glacier solution won’t be a good idea, unless I reduce a lot my backups frequency.

[Sectros (Sector Pag)]

where in the vps are the backups stored?

[pfaffman (Jay Pfaffman)]

@Sectros

where in the vps are the backups stored?

I added this to the OP:

@Discourse

If you’re self-hosted you can access them at /var/discourse/shared/standalone/backups/default.

[matenauta]

Can we chose the folder of the backups or there is a workaround without coding?

I’m using an storage data from my hosting provider so I can mount and use like local but it not supposed to be saved into default path.

[pfaffman (Jay Pfaffman)]

If you want it to be saved in a different place then you’d need to change that in your app.yml

[shyguy]

Automatic Backups on Backblaze B2

here’s how i have it set up for a hypothetical site hosted on example.com

1. make an account on backblaze (atm, no need to enter payment for <10GB which is free)
2. create a bucket (backblaze > B2 Cloud Storage)
   • name: $sitename-discourse-$random padded to 30char
     • in this example: example-discourse-g87he56ht8vg
     • discourse needs bucket name to be lowercase letters, numbers, and dashes only
     • i suggest keeping it 30 char or less since that shows up nicely in backblaze’s webui without wrapping
   • private bucket
   • enable encryption (SSE-B2)
   • enable object lock
3. create an application key (backblaze > account > app keys)
   • keyName: example-discourse
   • bucketName (Allow access to Bucket(s)): example-discourse-g87he56ht8vg
   • capabilities: read and write
   • leave namePrefix and validDurationSeconds blank
4. configure discourse B2 settings (discourse > admin > settings)
   • backup_location: s3
   • s3_backup_bucket: example-discourse-g87he56ht8vg
   • s3_endpoint: this is shown on the bucket page – make sure to prepend with https://
   • s3_access_key_id: (from previous step)
   • s3_secret_access_key: (from previous step)
     • backblaze only shows you the key once (at creation)!
   • btw, you can also set these as env vars in your container yml instead. this would let you restore with only that file and nothing else:

env:
  ## Backblaze B2 Backups
  # DISCOURSE_BACKUP_LOCATION: 's3' # uncomment to recover from cli
  DISCOURSE_S3_ENDPOINT: 'https://....backblazeb2.com'
  DISCOURSE_S3_BACKUP_BUCKET: 'example-discourse-g87he56ht8vg'
  DISCOURSE_S3_ACCESS_KEY_ID: '...'
  DISCOURSE_S3_SECRET_ACCESS_KEY: '...'
  # DISCOURSE_DISABLE_EMAILS: 'non-staff' # uncomment to disable email during a test restore
  ## you can restore with no data beyond this container yml.
  ## uncomment DISCOURSE_BACKUP_LOCATION above, build container (./launcher rebuild ...),
  ## and then run this inside container (it will restore from B2 bucket):
  ##   discourse enable_restore
  ##   discourse restore <example-com-...tar.gz> # choose restore filename by browsing B2 webui
  ## remember to disable restore afterwards

5. configure backup retention
   • discourse:
     • backup_frequency: 1 (daily backups in this example, but you could do weekly)
     • maximum_backups: disregard this setting – let backblaze handle it
     • s3_disable_cleanup: true (Prevent removal of old backups from S3 when there are more backups than the maximum allowed)
   • backblaze (go to your bucket’s settings):
     • Object Lock (Default Retention Policy): 7 days
     • Lifecycle Settings (custom):
       • fileNamePrefix: default/example-com (optional)
       • daysFromUploadingToHiding: 8 days
         • this should be object lock + 1
       • daysFromHidingToDeleting: 1 day

to summarize retention in this example:

• discourse creates backups every 1 day
• each backup file is immutable for 7 days after upload to B2 (object lock). this protects you against accidents, ransomware, etc.
• 8 days after upload, the object lock on the backup expires. since it’s mutable again, a lifecycle rule can hide the backup file
• the next part of the lifecycle rule deletes any file 1 day after it’s hidden

so you get daily backups. retention time is one week during which backups can’t be deleted no matter what. then backups are deleted 2 days later. so really a backup lives for 9 days or so.

hope that helps someone

---

on second thought, maybe it’s better to let discourse handle retention (maximum_backups). that way, your backups won’t automatically start expiring if discourse is down. you wouldn’t want a clock ticking on them while trying to recover. if you went that way, you could set maximum_backups=8 and s3_disable_cleanup=false in this example and not use a lifecycle policy in B2. you would still use the object lock policy (7 days), though.

edit: actually, i think you do still need a B2 lifecycle policy because i think files only get ‘hidden’ and not deleted when an S2 client deletes them. i’m using the “Keep only the last version of the file” policy, which is equivalent to daysFromHidingToDeleting=1, daysFromUploadingToHiding=null.

i guess think it over and decide which approach is right for you.

btw, i realize there’s some back in forth in this post. i think it’s informative as-is, but if someone wants, i could make another slightly simpler post with my actual recommendations.

[pfaffman (Jay Pfaffman)]

@shyguy

configure discourse B2 settings (discourse > admin > settings)

If you put those in environment variables as described in Configure an S3 compatible object storage provider for uploads then you can restore your site to a new server from the command line with only your yml file.

The rest seems like a good plan.

[shyguy]

@pfaffman

you can restore your site to a new server from the command line with only your yml

discourse restore <backup.tar.gz>

this will look in your bucket if you have the env vars set? pretty cool if so.

and in that case, you could probably also set them manually with export in bash in the unlikely event that you have to recover. that is, if you don’t want to keep secrets in your container yml for some reason.

[Crius]

@pfaffman

If you’re self-hosted you can access them at /var/discourse/shared/standalone/backups/default.

Just for confirmation, once I have moved to S3 backups and tested that they works, can I safely delete the content of that folder to reclaim used space?