I’m trying a few things on cloudflare and was thinking of leaving R2 in the future. That’s why, when I was examining the backups, I saw that the installations were not included and it gave an error. When making a backup manually, it gives the same error in the transaction logs. Pictures and files in local are backed up, but pictures in Cloudflare R2 are not backed up.
An example error code:
Failed to download original/1X/4b8754c367e54bc5271454a09b8bd4c1a8b882d5.png because Aws::S3::Errors::Http501Error
/var/www/discourse/lib/s3_helper.rb:280:in `rescue in download_file'
/var/www/discourse/lib/s3_helper.rb:277:in `download_file'
/var/www/discourse/lib/file_store/s3_store.rb:338:in `download_file'
/var/www/discourse/lib/backup_restore/backuper.rb:321:in `block in add_remote_uploads_to_archive'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:71:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:71:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:245:in `block in in_batches'
<internal:kernel>:187:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:229:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/relation/batches.rb:70:in `find_each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activerecord-7.0.8.4/lib/active_record/querying.rb:22:in `find_each'
/var/www/discourse/lib/backup_restore/backuper.rb:315:in `add_remote_uploads_to_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:248:in `create_archive'
/var/www/discourse/lib/backup_restore/backuper.rb:40:in `run'
/var/www/discourse/script/spawn_backup_restore.rb:9:in `backup'
/var/www/discourse/script/spawn_backup_restore.rb:31:in `block in <main>'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `fork'
/var/www/discourse/script/spawn_backup_restore.rb:4:in `<main>'
That is expected, and mostly the point of S3/R2 is that you don’t need to back the stuff up. But there is a hidden setting that will download all of the images from S3 and put them in the backup.
Oh, but you do have DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true.
So maybe R2 doesn’t work as originally suggested. I thought I’d tested and had it work, but I didn’t try downloading images to the backup and I likely didn’t try doing a restore or downloading a backup file.
This probably belongs in that other topic since it seems to contract the “maybe it works” hypothesis presented there.
This setting is active, but unfortunately it does not import the images from R2 when backing up. You can see this in the log when taking a manual backup.
I’m undecided and I’m thinking of leaving cdn and using local storage. In fact, this is expected to happen, in case the opinions change.
I’ll raise the issue again but it fills up with error codes with every automatic backup. If it is not backed up, I think it would be better not to request the files on S3 to avoid wasting time. This way we don’t fill the file with unnecessary error codes.
My guess is that it won’t let you download them because it’s trying to protect you from someone stealing all of your images. Sounds like another way that R2 isn’t compatible with discourse.
I’ve been on R2 for a long time. Everything looks good, there are just a few issues (like above) but they can be fixed. Over time, the community will work on this issue, until then, good luck to you
That’s interesting! I’ve only seen you complain about things that weren’t working. My quick tests looked like it worked, but I didn’t test extensively.
Some kind of rate limiting is my best guess on the downloads to backup, though I’d think there would be errors?
After your tests, we ran R2 and my problems with R2 were as follows:
When I turned off S3, the connections turned into strange R2 connections (what I saw when I looked in Docker). And when I tried to fix it and rebake, the images on the site were flying and distorted. After correcting it after a few tries, I set R2 only for upload and use it. So I haven’t found a way to get rid of it yet (or I couldn’t).
Another was that it produced an error message when the backup was made. Considering that there are 10 thousand images, it fills the error logs in vain. This can sometimes cause us to miss important problems. (It may be better if it does not give an error message, but it should be noted that the data on S3 will not be added to the backup)
I don’t remember having any problems other than these. If I want to back up the files on R2, I can connect via winscp and download them to my computer. If anyone calls, let’s leave a note here
It just occurred to me that those using S3 can turn off this setting “backup with uploads”. so we can get rid of the error message. Why didn’t I think of this before…
