Users kuaza and pfaffman discuss an issue where uploads stored in Cloudflare R2 are not being included in Discourse backups, despite the setting DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true being enabled. The backup process fails with an Aws::S3::Errors::Http501Error when attempting to download R2-hosted files, though local uploads are backed up successfully. pfaffman suggests this may be intentional — since S3/R2 is designed to be durable — and notes that R2 may not fully support backup downloads due to compatibility issues or rate-limiting. kuaza confirms the setting is active but ineffective and considers switching to local storage to avoid error logs cluttering backups. Later, kuaza realizes that disabling the backup-with-uploads setting would suppress the error, offering a workaround. Other issues with R2 mentioned include distorted images during rebake and unexpected Docker connections after disabling S3. kuaza remains optimistic that the community will resolve R2 compatibility over time, and suggests manual downloads via WinSCP as a temporary solution for backing up R2 files. The issue is noted as potentially belonging in the broader R2 setup discussion topic here.
I’m trying a few things on cloudflare and was thinking of leaving R2 in the future. That’s why, when I was examining the backups, I saw that the installations were not included and it gave an error. When making a backup manually, it gives the same error in the transaction logs. Pictures and files in local are backed up, but pictures in Cloudflare R2 are not backed up.
An example error code:
Failed to download original/1X/4b8754c367e54bc5271454a09b8bd4c1a8b882d5.png because Aws::S3::Errors::Http501Error
/var/www/discourse/lib/s3_helper.rb:280:in `rescue in download_file'
/var/www/discourse/lib/s3_helper.rb:277:in `download_file'
/var/www/discourse/lib/file_store/s3_store.rb:338:in `download_file'
/var/www/discourse/lib/backup_restore/backuper.rb:321:in `block in add_remote_uploads_to_archive'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:71:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:71:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:245:in `block in in_batches'
<internal:kernel>:187:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:229:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:70:in `find_each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/querying.rb:22:in `find_each'
/var/www/discourse/lib/backup_restore/backuper.rb:315:in `add_remote_uploads_to_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:248:in `create_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:40:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:31:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
That is expected, and mostly the point of S3/R2 is that you don’t need to back the stuff up. But there is a hidden setting that will download all of the images from S3 and put them in the backup.
Oh, but you do have DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true.
So maybe R2 doesn’t work as originally suggested. I thought I’d tested and had it work, but I didn’t try downloading images to the backup and I likely didn’t try doing a restore or downloading a backup file.
This probably belongs in that other topic since it seems to contract the “maybe it works” hypothesis presented there.
This setting is active, but unfortunately it does not import the images from R2 when backing up. You can see this in the log when taking a manual backup.
I’m undecided and I’m thinking of leaving cdn and using local storage. In fact, this is expected to happen, in case the opinions change.
I’ll raise the issue again but it fills up with error codes with every automatic backup. If it is not backed up, I think it would be better not to request the files on S3 to avoid wasting time. This way we don’t fill the file with unnecessary error codes.
My guess is that it won’t let you download them because it’s trying to protect you from someone stealing all of your images. Sounds like another way that R2 isn’t compatible with discourse.
I’ve been on R2 for a long time. Everything looks good, there are just a few issues (like above) but they can be fixed. Over time, the community will work on this issue, until then, good luck to you
That’s interesting! I’ve only seen you complain about things that weren’t working. My quick tests looked like it worked, but I didn’t test extensively.
Some kind of rate limiting is my best guess on the downloads to backup, though I’d think there would be errors?
After your tests, we ran R2 and my problems with R2 were as follows:
When I turned off S3, the connections turned into strange R2 connections (what I saw when I looked in Docker). And when I tried to fix it and rebake, the images on the site were flying and distorted. After correcting it after a few tries, I set R2 only for upload and use it. So I haven’t found a way to get rid of it yet (or I couldn’t).
Another was that it produced an error message when the backup was made. Considering that there are 10 thousand images, it fills the error logs in vain. This can sometimes cause us to miss important problems. (It may be better if it does not give an error message, but it should be noted that the data on S3 will not be added to the backup)
I don’t remember having any problems other than these. If I want to back up the files on R2, I can connect via winscp and download them to my computer. If anyone calls, let’s leave a note here
It just occurred to me that those using S3 can turn off this setting “backup with uploads”. so we can get rid of the error message. Why didn’t I think of this before…
