S3 Backups - not even trying?


(Howard Jones) #1

I’ve had a (rather empty) Discourse instance running from Docker on DigitalOcean for a few months, and backing up to its local disk. Last night I spent some time trying to figure out how to get it to do offsite backups. My preference would have been for Dropbox, which is pretty simple to set up, but the plugin for that has been withdrawn (?). So S3 it is.

I don’t care about file/image uploads to S3, just backups. So I created a new bucket, created a new IAM user in a new group, and gave that group permission to access my buckets (initially, I did create a second bucket for forum data in case I change my mind):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::weathermap-discourse-data",
                "arn:aws:s3:::weathermap-discourse-data/*",
                "arn:aws:s3:::weathermap-discourse-backups",
                "arn:aws:s3:::weathermap-discourse-backups/*"
            ]
        }
    ]
}

I plugged in the token and secret values from my new user to the Files settings page, and set the correct region. Then checked the box in the Backups page to copy backups to s3, and ran a manual backup. The logs don’t mention s3 at all! It appears to not even try to send data…

The descriptions on this site seem to be rather out of date (don’t really reflect the current AWS console), and also assume a level of knowledge with AWS I don’t have (e.g. what does “Use AWS EC2 IAM role to retrieve keys. NOTE: enabling will override “s3 access key id” and “s3 secret access key” settings.” mean? Surely it needs some credentials to do that?)

A ‘test’ button in the settings somewhere to verify my S3 settings independently of actually using them would be really useful.

How can I faultfind what’s actually happening here? No S3 logging at all has left me a little stuck.


(Howard Jones) #2

From my linux desktop, I am able to upload files using the same credentials, so I know that the AWS permissions are roughly OK. To test this, I did:

pip install awscli

aws configure
     (enter the same details you gave discourse)
touch testfile
aws s3 cp testfile s3://weathermap-discourse-backups

and I can see that 0-byte file in the AWS console. You can also check with the same cli tool:

aws ls s3://weathermap-discourse-backups
   2017-09-01 10:44:16          0 testfile

(Jeff Atwood) #3

Plugin was not withdrawn, there are parallel efforts to allow multi-target backups. You have to pick which one you want, the old one that was hard-coded to Dropbox, or the new one with the multi-target backups.


(Howard Jones) #4

The recent discussion I can find is here: Discourse Backups to Dropbox (Deprecated)

After a long confusing thread: “Well, it’s not working, so it is cancelled for now. At least the dropbox bit. Copy your backup files through the browser, or something.”

So how about that S3 logging?


(Jeff Atwood) #5

That was before the changes to it were reverted.


(Jay Pfaffman) #6

Yes, but there is a replacement. See Synchronizer-base for any backup provider

Discourse Backups to Dropbox (Deprecated) includes a link at the top to the new discourse-sync-to-dropbox plugin.


(Howard Jones) #7

I played a little more over lunch. S3 still doesn’t do anything or log anything positive or negative, and the new backup-to-anything-base+dropbox plugins broke the docker container rebuild, so after all, I got the deprecated one working in 3 minutes.


(Jeff Atwood) #8

Cool, yes, the old plugin should continue to work. Sorry for the confusion.