Hosted to self-hosted migration: past uploads still reference discourse infra

Product Support
17

I did what everyone told my not to do: to meddle with the database and its dumpfile. Currently, most everything works, except for same cases of:

<img src="https://forum.everviz.com/images/transparent.png"
alt="image" data-orig-src="upload://npqpp5O0wbL89nR9OXtP7Btu4hc.png"
width="517" height="90" style="aspect-ratio: 517 / 90;" loading="lazy">

Let’s compute the b62 and take its hex

npqpp5O0wbL89nR9OXtP7Btu4hc = 0x a411c90267cafca7a1cbcd7c8f4f9b8db17e51ba

Now try to find it from /var/www/discourse/public/uploads:

find . -name '*a411c90267cafca7a1cbcd7c8f4f9b8db17e51ba*'
./default/original/1X/a411c90267cafca7a1cbcd7c8f4f9b8db17e51ba.png

Yes!

932×164 18.5 KB

But why is it transparent.png in the post? I’ve done rake uploads:recover_from_tombstone and rake posts:rebake

How did I get here?

The uploads column in the database, for table url would still show cdck as part of the source URL for images. I dropped into the database from inside the container:

postgres psql discourse

Then

UPDATE uploads
SET url = REPLACE(
           url, 
           '//cdck-file-uploads-europe1.s3.dualstack.eu-west-1.amazonaws.com/standard21/uploads/everviz/', 
           '/uploads/default/'
         )
WHERE url LIKE '//cdck-file-uploads-europe1.s3.dualstack.eu-west-1.amazonaws.com/standard21/uploads/everviz/%';

This showed promising results, where most original images and thumbnails would reappear.

One step further: modifiying dumpfiles

The assumption is that Discourse is stateless* and the only thing we need to care about is what is inside the database. I was not eager to fiddle around with rake tasks or ruby for this, as I am not very familiar with either of those or Discourse internals. I just want results, fast.

*short of the public folder which contains our images. We can nevertheless confirm that we have everything we need.

So we download a copy of the database from the UI, then open it in VSCode and regressively substitute cdck (bucket) references and europe1 (bucket from CDN) references

By regressively I mean that in some instances you would see ‘//…’ and in other cases you would see ‘https://’. Therefore you need to match and replace ‘//…’ first, else you will have trailing https: across the file.

Then reupload the modified dumpfile. Part of what made this all tricky is the base62 step, which makes it a little harder to go from a raw representation to the actual image URL.