# Sidekiq daylight saving time issues?
**URL:** https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155
**Category:** Bug
**Created:** [1 Novembre 2015, 5:39pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155 "2015-11-01T17:39:07Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [1 Novembre 2015, 5:39pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/1 "2015-11-01T17:39:07Z")
Today we found a number of (independent) servers running into problems at the same time.
Symptoms: read only mode was enabled and the disk was full.
This turned out to be caused by the backup scheduler creating a _LOT_ of backups. The disk filled up so quickly that our monitoring didn’t even catch it in time.
Excerpt from sidekiq.log below.
This didn’t happen on all servers, but on about 8% of them. The issues started independently just after 2:00 AM EST, i.e. just when daylight savings time ended. I can’t explain why this is causing it, but I do think it’s related.
Happened both on 1.4.2 and on 1.5.0beta3 servers.
```
[STARTED]
'system' has started the backup!
Marking backup as running...
Making sure '/var/www/discourse/tmp/backups/db2155/2015-11-01-020020' exists...
Making sure '/var/www/discourse/public/backups/db2155' exists...
Writing metadata to '/var/www/discourse/tmp/backups/db2155/2015-11-01-020020/meta.json'...
...
(lots of backup logs)
...
Unpausing sidekiq...
Marking backup as finished...
Finished!
[SUCCESS]
[STARTED]
'system' has started the backup!
Marking backup as running...
Making sure '/var/www/discourse/tmp/backups/db2155/2015-11-01-020143' exists...
Making sure '/var/www/discourse/public/backups/db2155' exists...
Writing metadata to '/var/www/discourse/tmp/backups/db2155/2015-11-01-020143/meta.json'...
[STARTED]
'system' has started the backup!
Marking backup as running...
Making sure '/var/www/discourse/tmp/backups/db1714/2015-11-01-020143' exists...Enabling readonly mode...
Making sure '/var/www/discourse/public/backups/db1714' exists...
Writing metadata to '/var/www/discourse/tmp/backups/db1714/2015-11-01-020143/meta.json'...
Pausing sidekiq...
Waiting for sidekiq to finish running jobs...
Dumping the public schema of the database...
[STARTED]
'system' has started the backup!
Marking backup as running...
Making sure '/var/www/discourse/tmp/backups/db1714/2015-11-01-020143' exists...
Making sure '/var/www/discourse/public/backups/db1714' exists...
Writing metadata to '/var/www/discourse/tmp/backups/db1714/2015-11-01-020143/meta.json'..
```
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [1 Novembre 2015, 6:04pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/2 "2015-11-01T18:04:24Z")
Interesting problem.
If daylight savings was a “standard” it would be easy enough to not run back-ups during the affected time.
But it is anything but standardized
> **[Daylight saving time | Procedure](https://en.wikipedia.org/wiki/Daylight_saving_time#Procedure)**
>
> The relevant authorities typically schedule clock changes to occur at (or soon after) midnight and on a weekend, in order to lessen disruption to weekday schedules. A one-hour change is usual, but twenty-minute and two-hour changes have been used in the past. Notable exceptions today include Lord Howe Island with a thirty-minute change, and Troll (research station) that shifts two hours directly between GMT and CEST since 2016.
> In all countries that observe daylight saving time seasonally For a m...
Maybe some way of rate limiting back-ups based on when the last successful one was run?
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [1 Novembre 2015, 6:25pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/3 "2015-11-01T18:25:37Z")
We have learned to never schedule cronjobs between 2:00 AM and 3:00 AM because of this.
But then again, I can’t figure out how this causes Sidekiq to behave like this.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [1 Novembre 2015, 7:03pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/4 "2015-11-01T19:03:45Z")
Hmm, anything we can do to protect against time change by an hour causing this @zogstrip?
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [2 Novembre 2015, 4:47pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/5 "2015-11-02T16:47:07Z")
Move the automatic backup to 4AM?
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [2 Novembre 2015, 5:49pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/6 "2015-11-02T17:49:40Z")
That would be a good workaround, but doesn’t solve the root issue. Apparently something goes nuts - 2:00AM happening twice doesn’t explain why multiple backups started to run simultaneously. So there must be an additional bug in Sidekiq that got triggered somehow.
---
### Author:  [@Mittineague](https://meta.discourse.org/u/Mittineague)
#### Post date: [2 Novembre 2015, 6:02pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/7 "2015-11-02T18:02:20Z")
> [@michaeld](#):
>
> 2:00AM happening twice
Technically it doesn’t happen twice - IF - the timezone is used.
eg. 2 AM EST is not the same as 2 AM EDT
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [3 Novembre 2015, 7:20am UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/8 "2015-11-03T07:20:22Z")
> [@Mittineague](#):
>
> Technically it doesn’t happen twice - IF - the timezone is used. eg. 2 AM EST is not the same as 2 AM EDT
That is why I didn’t use the timezone in my sentence.
---
### Author:  [@eswald](https://meta.discourse.org/u/eswald)
#### Post date: [9 Novembre 2015, 7:14pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/9 "2015-11-09T19:14:04Z")
I’ve seen similar problems result from collecting the time from one source, but the time zone from another. This looks like the backup could be running at 2 AM EST, but getting saved as 2 AM EDT, then running again because it doesn’t think the 2 AM EST scheduled backup has run.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [10 Novembre 2015, 3:10am UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/10 "2015-11-10T03:10:11Z")
Can we add more protection here @zogstrip because this sounds quite painful.
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [12 Novembre 2015, 5:35pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/11 "2015-11-12T17:35:35Z")
This should fix it 🕒
[https://github.com/discourse/discourse/commit/3c2486e2baad698b6e7c2bebc6ed197111fed011](https://github.com/discourse/discourse/commit/3c2486e2baad698b6e7c2bebc6ed197111fed011)
---
### Author:  [@jomaxro](https://meta.discourse.org/u/jomaxro)
#### Post date: [4 Febbraio 2016, 8:29pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/12 "2016-02-04T20:29:10Z")
Was this resolved? Daylight Saving Time starts again in just over a month…
---
### Author:  [@michaeld](https://meta.discourse.org/u/michaeld)
#### Post date: [4 Febbraio 2016, 9:07pm UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/13 "2016-02-04T21:07:20Z")
It seems fixed, see the post by ZogStriP above yours. I think it only happened when the clock went back, so it shouldn’t occur in the spring anyway.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [5 Febbraio 2016, 12:55am UTC](https://meta.discourse.org/t/sidekiq-daylight-saving-time-issues/35155/14 "2016-02-05T00:55:17Z")