Help restoring - system hung at midnight

Rebake all posts matching a pattern

1 Like

Sadly that did not work, the URL didn’t change after Rebuilding the HTML and it still leading a

Oops! That page doesn’t exist or is private.

Any other ideas or thoughts?

Did rebuilding from the UX work or not?

Clicking on the “Rebuild HTML” button didn’t work. The link didn’t change and it still leads to the error page.

There’s a second issue I noticed after the restore. I had a look at the error logs and I noticed this. The link isn’t the same as the one in the post I rebuilt:
Failed to process hijacked response correctly : Errno::ENOENT : No such file or directory @ rb_sysopen - /XXXXX.s3.dualstack.us-east-1.amazonaws.com/optimized/1X/46728e07f9819907d1b18387bf02ea7fc25c7981_2_32x32.ico

The odd thing is that when I put the above URL into the browser it does actually serve up the icon.

Here’s the backtrace

Message (5 copies reported)

Failed to process hijacked response correctly : Errno::ENOENT : No such file or directory @ rb_sysopen - /XXXXX.s3.dualstack.us-east-1.amazonaws.com/optimized/1X/46728e07f9819907d1b18387bf02ea7fc25c7981_2_32x32.ico

Backtrace

/var/www/discourse/app/controllers/static_controller.rb:160:in read' /var/www/discourse/app/controllers/static_controller.rb:160:in block (2 levels) in favicon’
/var/www/discourse/lib/distributed_memoizer.rb:16:in block in memoize' /var/www/discourse/lib/distributed_mutex.rb:33:in block in synchronize’
/var/www/discourse/lib/distributed_mutex.rb:29:in synchronize' /var/www/discourse/lib/distributed_mutex.rb:29:in synchronize’
/var/www/discourse/lib/distributed_mutex.rb:14:in synchronize' /var/www/discourse/lib/distributed_memoizer.rb:12:in memoize’
/var/www/discourse/app/controllers/static_controller.rb:138:in block in favicon' /var/www/discourse/lib/hijack.rb:56:in instance_eval’

Then you didn’t need to figure out how to rebuild from the command line.

I’m not sure, but it sounds like it’s treating that as a filename rather than a bucket, but I’d need to look at the source to know for sure.

Any other ideas/thoughts on how to fix the /short-url broken links?

That happened to me as well, on an Oracle Cloud server. The kernel panicked, and so did I. I thought I was toast. But after about six or eight reboots from the cloud console, some of which were “pull-the-plug” reboots, and after about half-an-hour of waiting, the server came up long enough for me to edit grub.cfg and revert to the previous kernel.

I was able to save my instance thereby. A day later, there was another new kernel update offered, and that’s when I grew more certain my theory about kernel trouble was true. And I found the bug description to confirm it. Yeah, pretty nasty.

I devised a Stupid Grub Trick, as I call it, that I will try to find time to post anon, so one would be able to avoid such a calamity in the future.

Good luck with your restore, @RBoy. I must say this thread is giving me a queasy stomach after my own near disaster last – when was it, Wednesday?

By the way, you said you regained access to your old server. If you still have it or can get access once more – for me it took some hard reboots and some waiting – well, go in and upgrade one more time, as there is another new kernel that doesn’t have the bug. Or revert to the previous kernel.

1 Like