Bizarre Problems with migrate_from_s3

Dev
1

I was migrating from S3 and had some strange problems. First, shouldn’t the rake task be looking at cooked, and not raw? Or maybe this is a vestige of the old way images were put into raw?

https://github.com/discourse/discourse/blob/master/lib/tasks/uploads.rake#L126

The rest of this is a bizarre story. I don’t know whether it will help anyone. It’s another story that ends with “it got fixed by rebooting.”

But I decided to take matters into my own hands. The images had already been downloaded from S3 and manually put into uploads, so I fixed up the uploads like this:

bads=Upload.where("url like '//xxx-discourse%'")

bads.each do |up|
  up.url.gsub!(/\/\/xxx-discourse.s3-ap-southeast-1.amazonaws.com\//,'/uploads/default/')
  up.save
end

and then rebaked the posts like this:

bps = Post.where("cooked like '%//xxx-discourse%'")
bps.each do |post|
  post.rebake!
  post.save
end

But this didn’t fix all of them, somehow. After that, I still had a bunch of posts that still pointed to S3. I tried rebaking, but still there were some upload://serfsfsef.jpg that would rebake to the old S3 URL no matter what I did. And THEN I pasted that same URL into a new post, and it generated the right (not S3) URL. And then, I fixed up cook myself like this:

bps = Post.where("cooked like '%//xxx-discourse%'")
bps.each do |post|
  post.cooked.gsub!(/\/\/xxx-discourse.s3-ap-southeast-1.amazonaws.com\//,'/uploads/default/')
  post.save
end

And then, if I rebaked the post, it would still revert to the S3 URL.

And then, I rebooted the container, and it rebaked correctly. I guess something got cached somewhere, somehow.

4 Likes
2

That’s your answer right there :wink:

3

So perhaps we should change this:

https://github.com/discourse/discourse/blob/master/script/import_scripts/base/uploader.rb#L38

And, perhaps, matters are made worse because when the post is created the code doesn’t it’s embedding the old way so it’s not getting tagged as version 0 (1?), so the task that would run to fix it doesn’t know to? (I’ve not looked at that version code closely enough to know what I’m talking about, much less describe it cogently).