This topic later evolved into ‘Restore Failing’ from 4th post onwards, which is main issue now. You can ignore 1st 4 posts.
I’ve set my upoads to Aws S3 years ago/in the beginning only.
Even when I (as far as I can say) never enabled the option to include the S3 uploads in my backups, yet yesterday, when I chose to include ‘Uploads’ in my backups, I got this in my logs:
It raises a few curious discrepancies, which are baffling me:
Yesterday evening, I toggled on the option in my Admin Settings to include ‘Uploads’ into my Backups. Adn then when I viewed my local ‘Shared Uploads’ folder thru WinScp, it just had under 100 files in only 1x folder (no other 2x, 3x folders exist there, can share SS if needed). So why is backup logs showing around 3K files have been downloaded. (‘Failed to download’ is another headache/problem in these logs, but that is 'other). Now, if it is downloading these files from local storage, where are those so many files existing? And if it is downloading from S3, then a, why is it downloading from there, because I never changed that option in rails console to include s3 data in backups, nor did I create any similar option in the Env section of my yml file.
Then today I changed that option in Rails Console to ‘True’. And now, when I ran the backup job, it showed the same around 3.2k files downloaded, around 100 ‘Failed to download’. But when I checked in my Aws S3 bucket, it had nearly 10 times, 32k files of around 3gb. So why is it not downloading all those files?
Isn’t there a way to tally/synch all this data, and possibly, know what discrepancies are occuring and where?
Now I’m very much baffled, what should I do. My ultimate target is to shift my (too coslty) aws storage to some cheaper version (Hetzner itself, where my VPS is running, is very very inexpensive, so I can increase my primary server’s storage also).
Even when my ‘Uploads’ folder (in Aws S3 bucket) is more than 3_gb (3.2k files), why the backups are just under 1 gb (with just 2.9k files downloaded into the backup), even after enabling the option ‘include_s3_uploads_in_backup’ thru rails console?
That setting downloads the files to a temporary directory and includes them in the backup. It does not put them in the uploads directory. To get them in the uploads directory you need to restore the backup. I would do it on a new server so that if anything goes wrong your original server is still intact.
It sounds like some files may be missing. Do you have posts with missing images? Another possibility is that the uploads table includes uploads that are no longer referenced on posts, so those missing images don’t matter.
If it’s only 100 or so then it likely isn’t that big a deal.
Or it could be that a bug at some point cleaned up (deleted) files that should have been kept.
To see the files that it downloaded you’ll need to download the backup file and see what it includes. Restore your backup to a new server to see how it works.
But main point is why the backup is just under 1 gb (with just 3100 files) even when the S3 ‘Uploads’ folder only is of 3.2 Gb and 32K files. (backup logs clearly show that it downloaded around just 10%, 3k files).
[I find it very much cumbersome for me to create new discourse setup with a diff domain to test this thing, though I find it super easy to create a snapshot and then, if need be, revert my, very less busy, site within 5 min wo any fuss ]
Well, I thought even after changing the option in Rails C, why not add this line in yml also DISCOURSE_INCLUDE_S3_UPLOADS_IN_BACKUPS: true thinking that may be this would call all my uploads from Aws S3.
But when after changing this option in yml, rebuilt the container, ran the backup, I found the same lines in the backup logs (3000 media downloads with around 100 fails).
And when I tried to restore (I didn’t change any upload/S3 settings in my Admin settings yet), it gave error.
Full Log:
So, I disabled S3 uploads, and then tried to restore my 1 gb backup (backup is still on Aws S3), and it failed again. What can go wrong here at this?
Also, after failed restore I was signed out and when I signed in again, I was shown a banner that all non-staff emails are disabled. And when I tried to access the log from the link received on my email, that file isn’t found/inaccessible (my set error page is displayed).
When it was trying to restore, just before I was signed out, I was seeing these log msgs):
[2024-08-19 04:12:58] 'Bathinda_Helper' has started the restore!
[2024-08-19 04:12:58] Marking restore as running...
[2024-08-19 04:12:58] Making sure /var/www/discourse/tmp/restores/default/2024-08-19-041258 exists...
[2024-08-19 04:12:59] Downloading archive to tmp directory...
In subsequenty ‘FAILED-restore’ try, I was able to click on the log link just before I was signed out. Here it is: log- failed restore.txt (98.9 KB)
I’ve done some experimentation that new uploads are indeed being created on my local ubuntu server. But restoring them from S3 to local is failing. But the thing is that a few posts that I’ve checked are still continue to show the images from S3 (those aren’t missing).
Also, after ‘Failed Restore’, even when I sign in with the same admin, I cannot access the ‘Log.txt’ attachment. It shows page not available/my setup error page instead.
I actually tried to do exactly same as in the screen shots of yours in the post referred just above. Anyway, I’ll do now this only ‘Discourse restore’.
As I’ve understood/I hope, I don’t need to give any path to the backup (.tz) file lying anywhere, it’d automatically pick it up from my local server backup folder.
