Fixing or cleaning up broken links and assets after a restore

We had a server crash and had to create a new server and restore from a backup. It was a harrowing process, details for what happened and how we restored it here.

Now after restoring (the key was to disable S3 uploads), all the links to attachments in posts are broken (404 error). I’ve searched the forum and can’t find a solution and hoping someone can point me in the right direction.

I have two options

  1. Can I fix these broken short-url links which link to attachments embedded in posts (all the broken links are for attachments in posts; embedded images render fine, other internal link work fine)?

For example, the URL to the attachment on a post on the forum shows as https://XYZ.com/uploads/short-url/phu1HOLvkE8LWpkKYfnMPSWsvHh.zip This is what I’m seeing in the logs when I click on an attachment link in a post (which leads to a 404).

Message (5 copies reported)

Failed to process hijacked response correctly : Errno::ENOENT : No such file or directory @ rb_sysopen - /XXXXX.s3.dualstack.us-east-1.amazonaws.com/optimized/1X/46728e07f9819907d1b18387bf02ea7fc25c7981_2_32x32.ico

Backtrace

/var/www/discourse/app/controllers/static_controller.rb:160:in read' /var/www/discourse/app/controllers/static_controller.rb:160:in block (2 levels) in favicon’
/var/www/discourse/lib/distributed_memoizer.rb:16:in block in memoize' /var/www/discourse/lib/distributed_mutex.rb:33:in block in synchronize’
/var/www/discourse/lib/distributed_mutex.rb:29:in synchronize' /var/www/discourse/lib/distributed_mutex.rb:29:in synchronize’
/var/www/discourse/lib/distributed_mutex.rb:14:in synchronize' /var/www/discourse/lib/distributed_memoizer.rb:12:in memoize’
/var/www/discourse/app/controllers/static_controller.rb:138:in block in favicon' /var/www/discourse/lib/hijack.rb:56:in instance_eval’

I’m really hoping there’s a way to fix these short-url links after disabling the S3 upload option while restoring the server from a backup. A post re-bake did not fix it.

  1. If for someone reason this is a dead-end and cannot be fixed en mass, now I’ve had thousands of orphaned attachments on the S3 cloud, is there a way to clean them up and free up the space? Is there a way Discourse can go through it’s S3 upload bucket and clear out all orphaned assets?

It might be, or might have been, possible to figure out how to fix up those links, but figuring out how is beyond the scope of what’s feasible in a forum.

Upload.sha1_from_short_url('phu1HOLvkE8LWpkKYfnMPSWsvHh.zip')
=> "b13050bdcd2d58924ba6ab3e7608b16bfc3cd1b7"

See if you have a file called b13050bdcd2d58924ba6ab3e7608b16bfc3cd1b7.zip somewhere in your uploads and/or s3 bucket. If so, then it should be possible, though not easy, to fix things up.

Since you didn’t include the actual forum or bucket names, we can’t tell here.

Yes, I did find that under:

original/2X/b/b13050bdcd2d58924ba6ab3e7608b16bfc3cd1b7.zip

Happy to PM you the links/details

What’s very funny is that just the attachments (like files) are broken. Any embedded images display just fine.

Then the stuff is all there, you just need to re-write the posts somehow. Not sure why it’s not working, but the files are there.

Is there a way to run this en mass from the console or the UI or “download” these files from S3 to local?

I believe so, but I think that someone who’s familiar with discourse and rails will need to write it. I’m not aware of an existing solution to your problem. There are a few topics about moving between S3 buckets that might offer some clues, but I don’t think that your particular problem has been solved before.

After restoring, you should have re-enabled the S3 upload option. It sounds like you did not do that?

2 Likes

And it’s hard to do since it got set on the app.yml that might do it.

Would that fix the issue? Just wondering it’s safe to try.

Well, yes, it’s what I recommended you to do in the first place…
If you don’t re-enable S3 uploads then the short-url function will look for those uploads locally, but they are on S3.

2 Likes

I’ll try it but on the other hand when I enabled it it broke the restore (see my other topic). With Jays help I had to figure out how to disable the upload to eventually restore it. Have you been able to successfully restore a server with the option enabled?

You need to

  • Disable the setting
  • Restore
  • Enable the setting

As I described in our PM exchange last week

2 Likes