Configure automatic backups for Discourse

[codinghorror (Jeff Atwood)]

@pfaffman

You back up your servers once a week?

Yes, that is correct, for my self hosted servers.

If you prefer different settings, feel free to edit them from the defaults. What’s stopping you from doing so?

[pfaffman (Jay Pfaffman)]

@codinghorror

Yes, that is correct, for my self hosted servers.

Well, I’ll be!

@codinghorror

What’s stopping you from doing so?

I’m cleaning up topics like this one. If the defaults were changed, it wouldn’t be needed!

I’m convinced that the defaults won’t be changed, so I’ll move ahead. :wink

[codinghorror (Jeff Atwood)]

Once a week is a good starting point and a solid default.

[Discoursecurious]

@nathankershaw

@misterjtc

I am a bit confused with how to use this guide to configure backups as well.

I did what I thought made sense but I’ve gotten a little lost along the way and now I am getting “Failed to access backup storage: Access Denied”

I had this problem too, which was resolved by adding a couple of lines to the policy document:

"Resource": [
        "arn:aws:s3:::your-uploads-bucket",
        "arn:aws:s3:::your-uploads-bucket/*",
        "arn:aws:s3:::your-backups-bucket",
        "arn:aws:s3:::your-backups-bucket/*"
      ]

Should this be added to the OP (which isn’t a wiki)? Or perhaps in the OP of Setting up file and image uploads to S3?

I had the very same problem!

i suggest to add a note regarding this on the initial post too!

[jessicah (Jessica Hamilton)]

Would be nice if this could be used to upload to a different S3 compatible provider, such as running MinIO.

[pfaffman (Jay Pfaffman)]

If you want to use minio, see Using Object Storage for Uploads (S3 & Clones). If you want different services for backups and assets then you’re out of luck (though there are ways to set up triggers to copy from one bucket 8 another)

[jessicah (Jessica Hamilton)]

No, just want for backups, so they get automatically pushed to a different machine on my network.

[pfaffman (Jay Pfaffman)]

Then the guide I linked is what you’re looking for.

[Tris20 (Tristan)]

Is it possible to be more frequent than once per day? I’d rather not make assumptions and set backup freqeuncy to a float

[pfaffman (Jay Pfaffman)]

If you want more frequent backups you’ll need to do them externally.

[Tris20 (Tristan)]

Thanks for the hint. That pushed me towards the command line option which we can schedule to do whenever:

Backup discourse from the command line

These are some simple instructions on how to backup and restore your Discourse site from the command line. Backing up Enter the container: ./launcher enter app Run the backup: discourse backup exit This will print out the final filename of your backup. Download the backup and container configuration (if needed) to your local machine: scp =USER=@=HOST=:/var/discourse/shared/standalone/backups/default/=FILENAME= . scp =USER=@=HOST=:/var/discourse/containers/app.yml . Restoring First, …

[bksubhuti (Bhante Bhikkhu Subhuti)]

I got this to work, but it seems that the uploads checkbox was not really needed, nor do I understand the purpose. What is the purpose? The only thing I want is backups to s3 instead of local for my server. The server only has weekly automatic backups…

The Json also had problems… I was able to get it to work using another website reference. However, nobody could upload any images because I had the uploads checkbox checked (as described here)…Unchecking that box fixed the image upload problem for users and their profile pics.

What is the purpose of the images upload? I’m seriously hoping images are inside the s3 backups.
I had to do the instructions twice because I didn’t understand “uploads” and only made one bucket. Then I had to do it again with 2 buckets, and then I had to remove the checkbox for uploads. It might be good if there was a separate more simple topic for s3 backups … and only backups.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutLifecycleConfiguration",
                "s3:CreateBucket",
                "s3:PutBucketCORS"
            ],
            "Resource": [
                "arn:aws:s3:::classicaltheravadabucket",
                "arn:aws:s3:::classicaltheravadabucket/*",
                "arn:aws:s3:::classicaltheravadabackupbucket",
                "arn:aws:s3:::classicaltheravadabackupbucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

[pfaffman (Jay Pfaffman)]

@bksubhuti

It might be good if there was a separate more simple topic for s3 backups … and only backups.

Configure automatic backups for Discourse

So you’d like to automatically back up all your Discourse data every day? Go to the /admin settings, backup, and set backup_frequency to 1. [image] Now backup will be taken every day. Store backups on local server or Amazon S3 WARNING Storing backups and regular uploads in the same bucket and folder is no longer supported and will not work. By default backups are stored on the local server disk. In order to store backups on Amazon S3, you’ll need to create a unique, private …

Though I think that topic should be updated to recommend that the S3 configuration be moved to app.yml rather than the database, that way you can do a command line restore of the database with only the yml file and not have to configure it with a user and s3 confige before doing a restore.

[bksubhuti (Bhante Bhikkhu Subhuti)]

@Configure automatic backups for Discourse

WARNING Storing backups and regular uploads in the same bucket and folder is no longer supported and will not work. By default backups are stored on the local server disk. In order to store backups on Amazon S3, you’ll need to create a unique,

I’m not sure what you are talking about . My backups are working see picture
.
I use s3 because digital ocean backups are only weekly, and if the server crashes and gets deleted, it is not useful.
On the other hand. I’m hoping that restoring from s3 or downloaded s3 bucket will be fine.
I’m not uploading the images, and I am hoping that the s3 backups are being backed up including images (although very few).

[RGJ (Richard - Communiteq)]

@bksubhuti

s3 backups are being backed up including images

Generally: no.
It does not make much sense to backup images in an S3 bucket into another S3 bucket?

[bksubhuti (Bhante Bhikkhu Subhuti)]

can you be less ambiguous?
The instructions had 2 s3 buckets. I could not get that to work.
I have only one s3 bucket. Hopefully pictures are included in that backup… is that correct?

I would imagine that local backups work the same, Right?

Please answer in full sentences regarding my questions. The tutorial was very confusing as well.

[RGJ (Richard - Communiteq)]

What is ambiguous about “no” ? (And what’s not ambiguous about “backups being backed up” )

Let me try again.

If you have configured uploads to be on S3 then uploads are not included in your backup.

[bksubhuti (Bhante Bhikkhu Subhuti)]

Let’s use the term “pictures” instead of uploads even though it can be other media.
This way we don’t confuse the text content as an upload which I’m uploading to s3.

So the 62 MB, backup files on s3 as pictured and uploaded in this thread does not include pictures ?

So how do I make sure the backups have these?
Do the local backups have the pictures too?

When I configured the s3 for "uploads (of media) " which was ambiguous as well. Nobody could post pictures because they were rejected from s3…

Is there a way to have both local and s3 daily backups?
I could care less if 5 days of pictures were lost, we are mostly a text based group.
But I would care if 5 days of text were lost. Digital Ocean only does 7-day backups if you pay them.
So even though I can backup daily, if the droplet gets hacked or damaged, then we lose those backups… I’m starting to think there is not much added value in s3.

I wish there were simple backups similar to WordPress which lets me backup to my google or dropbox account.

[RGJ (Richard - Communiteq)]

@bksubhuti

Let’s use the term “pictures” instead of uploads even though it can be other media.
This way we don’t confuse the text content as an upload which I’m uploading to s3.

No, that is a bad idea, if you upload a text file as an attachment, it’s an upload as well, it will cause confusion. And text in a post is stored into the database. So I’m sticking to the term uploads.

If your uploads are on S3, they are not included in backups. In that case the backups only contain a copy of the database. It does not matter whether your backups are local or on S3.

If your uploads are not on S3, they are included in the backups. In that case the backups contain a copy of the database, and a copy of the uploads. It does not matter whether your backups are local or on S3.

@bksubhuti

So even though I can backup daily, if the droplet gets hacked or damaged, then we lose those backups

If you are storing something on S3, be it uploads or backups of the database, they will not get lost if your DO droplet gets hacked or damaged. So I don’t see your point.

Since your posts are about backups and not about file and image uploads, I’m moving these to another topic.

[Canapin (Coin-coin le Canapin)]

I’d like to automatically move my S3 Backups to Glacier but I’m confused by the steps linked in the first post, which doesn’t explain much, maybe because there’s outdated stuff.

Which options should be checked here?