Botched restore led to Sidekiq job failures

AI-generated summary

A user experienced Sidekiq job failures after a botched restore, receiving a VersionCheck error due to a Excon::Error::Timeout. The issue was traced to a network interruption at Discourse’s data centers affecting version checks, which was resolved as the backlog cleared. The user confirmed the VersionCheck worker returned to normal after processing queued tasks. Simsnet initially reported the problem, and gerhard clarified it was a minor, temporary issue unrelated to the restore. pfaffman suggested checking system resources, while Simsnet confirmed the problem was resolved.

Backed up my server prior to a mass-delete and a typo occurred (sleep comes first :blush:) so I ran a restore.

It quit halfway through the first time I ran it and I was never messaged me a log. I ran the restore a second time and it completed, this time with the messaged log.

Now, I’m getting the Sidekiq update error:
A check for updates has not been performed. Ensure sidekiq is running.

Tried triggering it manually from /sidekiq/scheduler but it keeps failing after 60 seconds.
The error in “retries” shows this error:
Jobs::HandledExceptionWrapper: Wrapped Excon::Error::Timeout: connect_write timeout reached

No clue what’s going on now. Forum is still up and a few other Sidekiq tasks are failing.

Maybe reboot. But likely you’re swamped running post restore processes. How much ram, disk, cpu, posts?

You can ignore that error. A network interruption at one of our data centers is currently affecting version checks for self-hosters. They will start working again when the incident has been resolved.

I’ve rebooted. The forum has a total of 100k+ posts (not sure where to pull an exact number) and there are several image hotlinks queuing every few minutes.

Other specs: 2 CPUs, 80GB storage, 4GB RAM

I saw you mention that in another post. Have other users hosting forums reported the issue as well?

No, they haven’t. It’s a minor issue and isn’t noticeable unless you stare at logs at the right time. :wink: It doesn’t affect Discourse at all. No need to worry.

All of the image hotlink tasks have pushed through and the Jobs::VersionCheck worker is now saying OK. Seems like that backlog kept it from working right.

@pfaffman @gerhard Thank you both very much!