Sidekiq processed 5 millions tasks then died? Have trouble running more than one day



I’m having trouble with Sidekiq since november 1st.
I’m on discourse 1.7b6 for a week now and everything was going great, but suddenly, it went nuts.

I didn’t have any notification between novermber 1st midnight and when I rebuilt the app on the 2nd. It then processed all notifications and mails and went inactive again.

What alarmed me is the incredible number of task processed by sidekiq since november 1st :

Yup, 1 million on november 1st, 4 millions on november 2nd. As you can see in the top left corner, 5 millions total, so most of the volume was processed over these two days.

What is strange is that it doesn’t appear in the failed/passed statistics :

(you can see the sidekiq tried to get back with overdued work)

I had to rebuild twice this morning (nov 3rd) to get the notifications to work again ( tested with a pm - green notification). It seems to work for now, even if all scheduled tasks are overdued. But I think it’s only dued to winter daytime Saving being effective in France since sunday.

I can’t tell if this is a specific problem due to daylight saving times or a bug, or my instance going nuts on halloween sugar… I thought I should document this issue, even if I can’t really categorize it. If you happen to know what could be behind it, I’d be very pleased to know what it is and how to avoid it in the future.

(Régis Hanol) #2

Do you have automatic backup enabled? Did one happen around Nov. 1st?

(Sam Saffron) #3

Very likely backup related, be sure you are running on tests-passed and run

cd /var/discourse
git pull
./launcher rebuild app

be sure to write cd in blue


the git pull said I was up to date.

As I manually make backups before maintenance operations, I just disabled automatic backups just in case.
I will inform you in case this happens again. Thank you.