S3 assets broken after moving forum

Support

AI-generated summary

Summary of S3 Assets Broken after Moving Forum

LimitlessGreen encountered issues with S3 assets after moving a Discourse forum. The problems included:

  • Broken preview images, working enlarged images
  • Working preview images, but no mouse-over for enlarging images
  • Broken Gravatars

The issue arose because S3 asset URLs were rewritten to internal ones after restoring the backup. LimitlessGreen manually downloaded the assets from S3 and moved them to the correct location.

It was suggested by michaeld that the issue might be due to misconfiguration, but LimitlessGreen clarified that the S3 configuration was overwritten during the restore process.

pfaffman mentioned that there is a hidden setting to include S3 assets in backups, but it requires access to the app.yml file, which LimitlessGreen did not have.

LimitlessGreen provided a step-by-step guide to reproduce the issue and is willing to perform SQL modifications to resolve the problem if guided.

The recommended solution is to re-run the restore process with the S3 credentials and then set the hidden setting to include S3 assets in backups. However, this is not possible in LimitlessGreen’s case due to the lack of S3 credentials.

1

Hi,

I had to move an abandoned Discourse forum. Luckily, we had multiple admins, so it was possible to do and download a backup. We used a S3 bucked for our assets, which was kind of a problem. I’ll make a feature request at this point:

Make it possible to include S3 assets into backups

I found out (not sure if it was before or after rebaking the posts), that S3 asset URLs got rewritten to internal ones. So I extracted all original and optimized S3 URLs out of the database file and downloaded them (kind of) manually. After moving them to the right place, all missing images got displayed. … It would be great to have an automatic feature for that.

So far, I encountered three bugs:

  1. Broken preview images, working enlarged images
  1. Working preview images, no mouse over for enlarging images
  • /t/microlongrange-openhd-edition/379 … I can only do 2 links on my first post
  1. Gravatars are broken (maybe unrelated):
    -
    grafik
    grafik396×438 14.6 KB

I found no problems on new posts.

Thanks for this great project,
Limitless Green

2

That sounds like a misconfiguration, not a bug. Did you configure the same S3 bucket on the new host before restoring?

3

I restored the backup, which also contains the configuration. After doing that (and force https) I don’t got pictures from the S3 bucket at all.

Now I removed the S3 configuration completely.

Isn’t it a complete configuration overwrite?

4

Except the S3 configuration, if it was previously configured using ENV vars in app.yml …

5

There is a hidden setting that does just that. Cdck turns it on for customers that they know plan to plan their hosting.

S3 configuration is best done in environment variables as described in Using Object Storage for Uploads (S3 & Clones).

6

After restoring, I had set configs in: s3 upload bucket and s3 cdn url.
I don’t have access to the app.yml (of the old server).

7

Oh, nice! :grinning: Where?
The only suitable option in the backup section I fond was:
grafik
(which was ticked)

I’m not sure if it’s clear: I don’t want S3 for now. And I had to get the assets from there. No one knows, how long this server will stay. :sweat_smile:

8

If you don’t have access to the server via ssh you can’t change the hidden setting.

9

Alright. And what should be done now in a case like mine?
What can lead to this broken image preview?

10

If it’s not too late (that is, there’s traffic on the forum that’s happened and you don’t want to lose) and you have the S3 credentials, then I’d probably put the s3 credentials in the app.yml and re-run the restore. From there, you could probably set the hidden setting, do a backup, and restore it on a server without S3.

But I think that’s not the case. And it’s not clear just what the issues might be since even you don’t remember exactly what you did when.

If stuff is missing (and you have s3 credentials), I might try using some S3 tool to pull down all of the uploads and then hack at the database to get it to point to them. But I am not good enough at that to explain here how to do it.

11

Yeah, sadly I don’t have the S3 credentials :frowning:

I can reproduce the steps:

  1. Installed a new Discourse Forum with discourse_docker behind Traefik
  2. Did a backup of the old forum (including some old assets)
  3. Restored the backup on the new forum
  4. Forced HTTPS
  5. Got asset URLs out of the dump.sql (from the backup)
  • egrep -o 'discuss-openhdfpv.eu-central-1.linodeobjects.com\/original\/1X\/[0-9a-z_]{0,}\.[0-9a-zA-Z]{0,}' ./dump.sql >> original.txt
  • egrep -o 'discuss-openhdfpv.eu-central-1.linodeobjects.com\/optimized\/1X\/[0-9a-z_]{0,}\.[0-9a-zA-Z]{0,}' ./dump.sql >> optimized.txt
  1. Downloaded the assets: cat optimized.txt | parallel --gnu "wget -nc -P ./optimized/1X/ {}" (same with original)
  2. Removed the S3 configurations
  3. rake posts:rebake

I can do SQL stuff at the database, when I know what to look for.