We’ve been building, testing and intentionally finding gaps in our Discourse forum before we push it entire thing to production. I just migrated my test forum that had S3 uploads enabled from one server to another and upon restoration all of the URLs for anything attached got rewritten to the forum’s URL and not the S3…
Fortunately this is a test forum so we don’t really care too much about the data, however I’d like to:
A. Still fix this.
B. Find a way to mitigate/prevent this from happening in prod.
It wasn’t just posts but it was all images/media/content/avatars(which is pretty dramatic) that got affected by this…
Before restoring you can configure S3 on the destination site in your app.yml (not via the admin dashboard), once you confirm that’s setup and can reach the correct bucket you can go ahead with the restore and media should be linked up correctly.
We effectively did that and that’s how we ended up in that situation.
I copied the app.yml as it was to the destination and then backed up from the original. It was the restoration that was the problem as when we restored, that’s when the URLs got rewritten despite the fact that nothing had changed and we still had S3 uploads enabled.
We did eventually resolve this with a rebake(we think, Discourse’s caching is very brutal so between the many solutions we tried, we don’t actually know what did it), however the questions remain unanswered as to how we do migrations with minimal issues or even restoring from a back up if we have to on prod.
That sounds like you did configure S3 via site settings, not env variables as Kris suggested. The restore process needs to know about S3. That’s not possible with site settings.
You could also create a backup without uploads on the console of you want: discourse backup --sql_only
Restoring such a backup won’t rewrite upload URLs. So, as long as your new server has access to the same S3 bucket, this works.
The S3 configuration is in the app.yml. Not site settings.
Edit:
I realize I’m not being overly explanative and I’m not meaning to hide details.
We use OVH S3 and it’s configured in app.yml.
I backed up our test forum without uploads but still had S3 enabled at that point.
I then restored it on the new site with the exact same app.yml and that’s where the problem started. To be clear it’s fixed right now, but I’m unsure if it was me rebaking it several times or discourse caching aggressively. Which is why I need to know how to really do this properly and get it right first time. My fear is that if I ever need to restore a back up to our production instance and we run into this, I need to know exactly how to fix this asap before users notice.
As I said, if you want to restore on a server that uses the same S3 bucket, make sure to have S3 configured in app.yml and create a backup without uploads (discourse backup --sql_only). Upload URLs won’t be rewritten when the backup doesn’t contain uploads.
If you wish to restore on a server that uses a different S3 bucket, or no S3 config at all, use a full backup with uploads. Upload URLs will be rewritten during restore.
Are you 100% sure that you configured the OVH S3 via environment in app.yml on both servers and used a backup without uploads (.sql.gz file extension)?
When I initially restored it with the uploads done it actually broke so I had to completely wipe and start again this time, taking a backup without uploads. That’s where the issue started. The URLs were still written incorrectly.
I’m not sure how that happened. The restore process skips all upload-related code (including rewriting of upload URLs) when you restore a .sql.gz file.
Maybe we are talking about different things? I’m talking about the url column in the uploads table, which usually is //your-s3-bucket/original/... vs /uploads/original on local.
One caveat of restoring a .sql.gz file is that no URLs are rewritten at all. It expects that the server is reachable via the same hostname as the server where the backup was created. You will need to remap URLs if you change hostnames.
No hostnames were changed. Just updated A records, backed up and done.
User avatars were missing(and that was because I did not migrate the uploads folder). S3 images for attachments/media was re-written for the forum URL. Not the bucket URL.
So whilst I was expecting URLs for S3 uploads for the above to be written as https://some-bucket-name-here.s3.bhs.io.cloud.ovh.net/optimized it was actually https://forum.somedomainhere.com/uploads/optimized which of course isn’t gonna work.
I can literally spin up another VM again and do a straight up restore if you want me to verify all of the steps I’ve taken.
I restored it by just restoring from the sql.gz that was stored in S3, and so far the only thing that broke was some user avatars and some posts, but I guess that was due to them either not being uploaded to the S3 when they were created.
In a production environment, assuming everything goes well from launch and everything is in S3… if I restore from a back up, I won’t have the same weird issue as from the opening post is that correct?
