Rebaking old posts won't pull new S3 CDN URL after S3 bucket rename

installation
1

After installing Discourse (2.7.0.beta1) and importing old Google Groups posts, I added S3 bucket/key configuration (but no CDN URL) and ran

rake uploads:migrate_to_s3

which seemed to be working great. All the images were uploaded to S3 and Discourse was trying to access them via an automatically generated S3 URL-- something like

https://ortus-discourse.s3.dualstack.us-west-2.amazonaws.com/original/1X/75747ca17a3ca01f298f836691e1990916bafccb.png

Then I renamed the bucket (to ortus-discourse-uploads) and setup a Cloudfront distribution in front of it with a Cloudflare-configured CNAME called

https://communitycdn.ortussolutions.com/

The cname, cloudfront, and S3 are all working great. URLs like this correctly serve up one of the images in the bucket:

https://communitycdn.ortussolutions.com/original/1X/75747ca17a3ca01f298f836691e1990916bafccb.png

However, the issue is that Discourse is completely stuck on the old ortus-discourse.s3.dualstack.us-west-2.amazonaws.com domain (which doesn’t work due to the bucket being renamed) and no matter how many times I’ve rebuilt the container or rebaked the old posts, I cannot for the life of me get Discourse to use the new CDN URL. I’ve Googled for a day on this and rebaked probably a dozen times (inside the app container) with various settings. Every forum thread just gives the same advice-- rebuild and rebake, but it isn’t working.

It’s not just the images in posts that are broken, even the <link rel="icon" type="image/png" href=""> and the site logo are stuck on the old domain and won’t use the S3 CDN URL.

Here are my current S3 settings.

  DISCOURSE_S3_ACCESS_KEY_ID: '********'
  DISCOURSE_S3_SECRET_ACCESS_KEY: '******'
  DISCOURSE_BACKUP_LOCATION: 's3'
  DISCOURSE_ENABLE_S3_UPLOADS: true
  DISCOURSE_S3_BUCKET: 'ortus-discourse-uploads'
  DISCOURSE_S3_UPLOAD_BUCKET: 'ortus-discourse-uploads'
  DISCOURSE_S3_BACKUP_BUCKET: 'ortus-discourse-backups'
  DISCOURSE_S3_REGION: 'us-west-2'
  DISCOURSE_S3_CDN_URL: https://communitycdn.ortussolutions.com

  DISCOURSE_CDN_URL: https://community.ortussolutions.com

I’ve even tried remapping the old URL to the new URL in posts like so

rake posts:remap["ortus-discourse.s3.dualstack.us-west-2.amazonaws.com","communitycdn.ortussolutions.com"]

but the command said 0 posts affected.

Nowhere in any of my env vars or DB settings do I have anything referencing the old bucket name of ortus-discourse so I can’t figure out where Discourse keeps getting that from. I’m new to Discourse and not a Ruby dev so I haven’t looked any deeper than what I can see in my app.yml, the admin UI, and the output of the rake commands I’ve found on the forums.

2

What is the value of those uploads on the Uploads table?

./launcher enter app
rails c
Upload.all.sample(10).pluck(:url)
3

@Falco Thanks for the reply. Here is the output of that command.

root@discourse-app:/var/www/discourse# rails c
Upload.all.sample(10).pluck(:url)
[1] pry(main)> Upload.all.sample(10).pluck(:url)
=> ["//ortus-discourse.s3.dualstack.us-west-2.amazonaws.com/original/1X/52b3aff4e63a7e38bef42d469bafd1ed7c1cc1a2.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/9f90374a280a4681332bcd2191b8de43462f8776.png",
 "//ortus-discourse.s3.dualstack.us-west-2.amazonaws.com/original/1X/29691fba566fc998a966aa93859753e3cf0b8528.cfc",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/6ae912ced40d60adc1356c1d7acf144b0fa0985a.jpeg",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/4dfe5b48fc8cb5d79880d70355c34d7ed02be812.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/897c4b4e755c1c8e93224a27187dc631a02e4388.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/59886f322e6834b567d473138108fab6e0f33764.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/e3e3429d63155cf0d850e161846d187bc6f273ea.jpeg",
 "//ortus-discourse.s3.dualstack.us-west-2.amazonaws.com/original/1X/1b701869b4b235daa8d6a9a7728766f3b4e69814.txt",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/c83aaee941462d47ef91f0c6448257d07487b231.png"]
[2] pry(main)>
4

So there are quite a few files with the old bucket indeed.

The remap you need should be:

./launcher enter app
rails c
DbHelper.remap("ortus-discourse.s3.dualstack.us-west-2.amazonaws.com", "ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com")

Please take a backup before, as there is no undoing of such operation.

2 Likes
5

@Falco Thanks again. I ran two remaps

DbHelper.remap("ortus-discourse.s3.dualstack.us-west-2.amazonaws.com", "communitycdn.ortussolutions.com")
DbHelper.remap("ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com", "communitycdn.ortussolutions.com")

I rebuilt the image and that fixed things like the site header logo. I’m rebaking all the posts again to see if it fixes the post images, but this will take a while.

So while we wait on the rebake, what went wrong here? Is this a bug in Discourse that my uploads got stuck on the old URL and wouldn’t change over to use the new one?

I read posts like this one that talked about changing a CDN URL, but they only mentioned replacing strings in the posts (which didn’t work) and didn’t mention the DBUtil remap at all. How do I change the legacy CDN URLs of images in posts?

I even tried re-running the import to s3 rake script, but received an error (sorry, I didn’t make a note of it at the time).

I even found a migrate_from_s3 rake script I nearly ran to see if I could start from scratch, but then I found a forum post here indicating it would ruin my database if I ran it so I left it alone.

I’m at a loss for what I should have done differently or what forum post would have answered this for me. (I really did try hard to solve this on my own before posting here!)

6

Sadly, the rebake does not seem to have fixed the images embedded into posts. What’s interesting is if I pick an old post and edit it, I see the image represented with something like so:

![COMMANDBOXERROR.png|1169x984](upload://yTDVQSa4wbIeLGEZvE7muXe8sAJ.png)

but when viewing the post, there is just a big empty space in the post that points to

https://community.ortussolutions.com/images/transparent.png

This is a somewhat recent change. Most of yesterday these images simply pointed to the old incorrect S3 URL, but at some point last night or today the transparent png made an appearance.

7

Hmmmm that is not what I told you to do on my reply :face_with_raised_eyebrow:

On the uploads table we expect the S3 location, and it’s only replace to the CDN on the cook process of the markdown.

You put the CDN in the Uploads table, and it’s not what the software does normally.

8

Sorry about that, I assumed you just didn’t read the OP carefully enough to see I mentioned the cloudflare distribution and it I needed to massage it to use my actual URL. I didn’t realize that the unwanted URL was actually stored anywhere in the DB. I nearly replied and asked if that’s what you meant first, but it seemed like it was obvious what I needed to do.

No worries, I can easily remap those back to the S3 URL. This is a new discourse install and the only uploaded content is all in the same place so easy to switch.

My questions about what went wrong in my previous post still stand.

9

I believe it’s as simply as that we don’t support changing the Object Storage bucket.

“Support” here meaning that you can’t change it and expect it to work, and there is no written guide nor pre-packaged rake task for it. So if you end up needing to change a bucket some database fiddling is necessary.

10

Ok, fair enough. So the thing I did that was the big issue was when I renamed the S3 bucket and assumed updating the name of the bucket in the admin would take care of everything I need. Perhaps a warning in the admin UI when editing the bucket name would be helpful here. (I started with the settings in the admin UI before I switched to using env vars, thinking that may somehow help) It certainly wasn’t clear that changing the bucket name after the upload would be an issue.

I’ve remapped the domains back to the correct AWS S3 domain.

DbHelper.remap("communitycdn.ortussolutions.com", "ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com")

[1] pry(main)> Upload.all.sample(10).pluck(:url)
=> ["//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/fc05f9be9b783479819fec68b1d8e493110007a4.cfc",
 "/images/d-logo-sketch-small.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/50f0d6f260cdb4ef91e29023d92b46df096ab34e.cfc",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/65d80cddc6dc15b9a4d1b9e9d88cc9a8928c5316.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/a86aa2a12183428f3289caa95787ea16f22e2e4d.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/9f76b5238b147a60c8ad5f65bd7fa4bb6b58d852.png",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/6a4c9b992e6cd8a15ddeaec0d158ebd473164525.zip",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/3a532dec6390d5087ed6154fc0335c2c0f1ea543.zip",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/93420848249ecea2261d405e46f7f450cc02a3af.txt",
 "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/d67c39e06ce624b9deb7625dd041d21cffd96df9.png"]

I’ve rebuilt the app again and now I’m rebaking the posts. I’ll see if the 37th try is a charm :slight_smile:

1 Like
11

All uploads have the proper S3 bucket URL, container is rebuilt, and all 30k posts are re-baked. I’m still seeing

/images/transparent.png

in place of all my images.

Editing the posts still shows this:

![COMMANDBOXERROR.png|1169x984](upload://yTDVQSa4wbIeLGEZvE7muXe8sAJ.png)

What’s interesting is that other attachments such as zip files are working fine now.

Is there a missing piece necessary to get the embedded images in old posts working?

12

Can you print the attributes of the Upload object correspondent to yTDVQSa4wbIeLGEZvE7muXe8sAJ ?

13

I would love to, but I’m going to need some help with this one. I’m a developer, but not a Ruby dev. I’ve figured out how to print out all the attributes of a random upload like so:

[17] pry(main)> Upload.all.sample(1)
=> [#<Upload:0x00005633230f8af0
  id: 353,
  user_id: 273,
  original_filename: "helloWorldF.zip",
  filesize: 50542,
  width: nil,
  height: nil,
  url: "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/3a532dec6390d5087ed6154fc0335c2c0f1ea543.zip",
  created_at: Wed, 30 Dec 2020 18:46:31 UTC +00:00,
  updated_at: Wed, 30 Dec 2020 18:46:31 UTC +00:00,
  sha1: "3a532dec6390d5087ed6154fc0335c2c0f1ea543",
  origin: nil,
  retain_hours: nil,
  extension: "zip",
  thumbnail_width: nil,
  thumbnail_height: nil,
  etag: nil,
  secure: false,
  access_control_post_id: nil,
  original_sha1: nil,
  animated: nil,
  verification_status: 1>]

And I’ve found some references for ROR’s active record for finding specific records but I can’t find any data in there that looks remotely like yTDVQSa4wbIeLGEZvE7muXe8sAJ. How do I find the corresponding upload record?

14

Ooh, good news. While researching how to find the Upload records, I came across this post which showed me how to convert the base62 string to the sha1 hash.

It mentioned images that just show the transparent.png had been tombstoned. I’m not entirely sure what that means, but I assume a process ran overnight while the images were broken and marked them as not in use. I was able to run

rake uploads:recover_from_tombstone

and it seems all my embedded images have came back and are now pointing to my S3 CDN!

And for what it’s worth, here how I found the Upload record starting with the yTDVQSa4wbIeLGEZvE7muXe8sAJ string.

[14] pry(main)> Base62.decode("yTDVQSa4wbIeLGEZvE7muXe8sAJ").to_s(16)
=> "f49428d6af35d7e0414408ccb65e7316f5003215"
[15] pry(main)> Upload.where( "original_filename ilike '%f49428d6af35d7e0414408ccb65e7316f5003215%'" )
=> [#<Upload:0x000056313aa91fe8
  id: 899,
  user_id: 549,
  original_filename: "f49428d6af35d7e0414408ccb65e7316f5003215.png",
  filesize: 25514,
  width: 1169,
  height: 984,
  url: "//ortus-discourse-uploads.s3.dualstack.us-west-2.amazonaws.com/original/1X/f49428d6af35d7e0414408ccb65e7316f5003215.png",
  created_at: Tue, 12 Jan 2021 23:01:35 UTC +00:00,
  updated_at: Tue, 12 Jan 2021 23:01:36 UTC +00:00,
  sha1: "f49428d6af35d7e0414408ccb65e7316f5003215",
  origin: nil,
  retain_hours: nil,
  extension: "png",
  thumbnail_width: 594,
  thumbnail_height: 500,
  etag: "6977f35ddbf39a4399dc76f92a5079d4",
  secure: false,
  access_control_post_id: nil,
  original_sha1: nil,
  animated: nil,
  verification_status: 1>]

Thank you again for your assistance @Falco I’m new to Discourse but you’ve proven very patient and helpful :+1:

3 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.