S3 assets broken after moving forum

Support
1

Hi,

I had to move an abandoned Discourse forum. Luckily, we had multiple admins, so it was possible to do and download a backup. We used a S3 bucked for our assets, which was kind of a problem. I’ll make a feature request at this point:

Make it possible to include S3 assets into backups

I found out (not sure if it was before or after rebaking the posts), that S3 asset URLs got rewritten to internal ones. So I extracted all original and optimized S3 URLs out of the database file and downloaded them (kind of) manually. After moving them to the right place, all missing images got displayed. … It would be great to have an automatic feature for that.

So far, I encountered three bugs:

  1. Broken preview images, working enlarged images
  1. Working preview images, no mouse over for enlarging images
  • /t/microlongrange-openhd-edition/379 … I can only do 2 links on my first post
  1. Gravatars are broken (maybe unrelated):
    -grafik

I found no problems on new posts.

Thanks for this great project,
Limitless Green

2

That sounds like a misconfiguration, not a bug. Did you configure the same S3 bucket on the new host before restoring?

3

I restored the backup, which also contains the configuration. After doing that (and force https) I don’t got pictures from the S3 bucket at all.

Now I removed the S3 configuration completely.

Isn’t it a complete configuration overwrite?

4

Except the S3 configuration, if it was previously configured using ENV vars in app.yml …

1 Like
5

There is a hidden setting that does just that. Cdck turns it on for customers that they know plan to plan their hosting.

S3 configuration is best done in environment variables as described in Using Object Storage for Uploads (S3 & Clones).

1 Like
6

After restoring, I had set configs in: s3 upload bucket and s3 cdn url.
I don’t have access to the app.yml (of the old server).

1 Like
7

Oh, nice! :grinning: Where?
The only suitable option in the backup section I fond was:
grafik
(which was ticked)

I’m not sure if it’s clear: I don’t want S3 for now. And I had to get the assets from there. No one knows, how long this server will stay. :sweat_smile:

8

If you don’t have access to the server via ssh you can’t change the hidden setting.

9

Alright. And what should be done now in a case like mine?
What can lead to this broken image preview?

10

If it’s not too late (that is, there’s traffic on the forum that’s happened and you don’t want to lose) and you have the S3 credentials, then I’d probably put the s3 credentials in the app.yml and re-run the restore. From there, you could probably set the hidden setting, do a backup, and restore it on a server without S3.

But I think that’s not the case. And it’s not clear just what the issues might be since even you don’t remember exactly what you did when.

If stuff is missing (and you have s3 credentials), I might try using some S3 tool to pull down all of the uploads and then hack at the database to get it to point to them. But I am not good enough at that to explain here how to do it.

11

Yeah, sadly I don’t have the S3 credentials :frowning:

I can reproduce the steps:

  1. Installed a new Discourse Forum with discourse_docker behind Traefik
  2. Did a backup of the old forum (including some old assets)
  3. Restored the backup on the new forum
  4. Forced HTTPS
  5. Got asset URLs out of the dump.sql (from the backup)
  • egrep -o 'discuss-openhdfpv.eu-central-1.linodeobjects.com\/original\/1X\/[0-9a-z_]{0,}\.[0-9a-zA-Z]{0,}' ./dump.sql >> original.txt
  • egrep -o 'discuss-openhdfpv.eu-central-1.linodeobjects.com\/optimized\/1X\/[0-9a-z_]{0,}\.[0-9a-zA-Z]{0,}' ./dump.sql >> optimized.txt
  1. Downloaded the assets: cat optimized.txt | parallel --gnu "wget -nc -P ./optimized/1X/ {}" (same with original)
  2. Removed the S3 configurations
  3. rake posts:rebake

I can do SQL stuff at the database, when I know what to look for.