# Backups have started failing due to server time being wrong
**URL:** https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802
**Category:** Support
**Created:** [February 11, 2019, 3:34am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802 "2019-02-11T03:34:42Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@modius](https://meta.discourse.org/u/modius)
#### Post date: [February 11, 2019, 3:34am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/1 "2019-02-11T03:34:42Z")
Out of the blue our backups have started failing. I’ve updated to the very latest version of Discourse and still have the same issue. Everything else appears to be working correctly.
System reports the following backup log.. which I’m hoping someone might be able to explain:
```plaintext
BackupRestore::BackupStore::StorageError
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:70:in `rescue in unsorted_files'
/var/www/discourse/lib/backup_restore/s3_backup_store.rb:58:in `unsorted_files'
/var/www/discourse/lib/backup_restore/backup_store.rb:21:in `files'
/var/www/discourse/lib/backup_restore/backup_store.rb:26:in `latest_file'
/var/www/discourse/app/jobs/scheduled/schedule_backup.rb:11:in `execute'
/var/www/discourse/app/jobs/base.rb:137:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:127:in `block in perform'
/var/www/discourse/app/jobs/base.rb:123:in `each'
/var/www/discourse/app/jobs/base.rb:123:in `perform'
/var/www/discourse/app/jobs/base.rb:185:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.8.1/lib/mini_scheduler/manager.rb:81:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.8.1/lib/mini_scheduler/manager.rb:29:in `block in initialize'
```
Going to the web admin screen for backups gives me this error… which is not much to go on:
Are there any other logs I should be looking at to shed light on the problem?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [February 11, 2019, 7:51am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/2 "2019-02-11T07:51:42Z")
My guess is you have weird S3 permissions / buckets set up?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [February 11, 2019, 1:16pm UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/3 "2019-02-11T13:16:05Z")
Do you have uploads and backups in the same bucket?
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [February 11, 2019, 3:06pm UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/4 "2019-02-11T15:06:32Z")
- Check `/logs` for errors or click on the `/admin/backups.json` link in the error page to get additional details about the error.
- I assume you are using S3 for your backups:
---
### Author:  [@modius](https://meta.discourse.org/u/modius)
#### Post date: [February 13, 2019, 5:50am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/5 "2019-02-13T05:50:07Z")
> [@gerhard](#):
>
> Check `/logs` for errors or click on the `/admin/backups.json` link in the error page to get additional details about the error.
Weird.. could have sworn i tried to click that link before. It yields more mystery 😉
```plaintext
{"errors":["The difference between the request time and the current time is too large."]}
```
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [February 13, 2019, 7:16am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/6 "2019-02-13T07:16:03Z")
Is the server clock time off?
---
### Author:  [@modius](https://meta.discourse.org/u/modius)
#### Post date: [February 14, 2019, 10:44am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/7 "2019-02-14T10:44:52Z")
> [@codinghorror](#):
>
> Is the server clock time off?
Well I tried the following:
```plaintext
$ date
Thu Feb 14 05:32:16 EST 2019
$ ntpdate ntp.ubuntu.com
14 Feb 05:26:58 ntpdate[12447]: step time server 91.189.91.157 offset -408.660323 sec
$ date
Thu Feb 14 05:27:02 EST 2019
```
So that was promising… but backup view still dead with the same error. So then I restarted the app:
```plaintext
$ ./launcher restart app
+ /usr/bin/docker stop -t 10 app
app
starting up existing container
+ /usr/bin/docker start app
app
```
Still no dice 🎲 🎲
```plaintext
$ reboot
The system is going down for reboot NOW!
```
It’s a Digital Ocean droplet running vanilla Ubuntu. Comes back up… still not working and the time offset is gone:
```plaintext
$ ntpdate ntp.ubuntu.com
14 Feb 05:36:42 ntpdate[2375]: adjust time server 91.189.91.157 offset -0.001713 sec
```
I was convinced the 5 minute time correction would be key. Is it possible it’s just stuck now?
Any other ideas?
* * *
> [@pfaffman](#):
>
> Do you have uploads and backups in the same bucket?
No. Separate bucket on S3 for backups; set-up unchanged for as long as anyone can remember.
* * *
> [@gerhard](#):
>
> Check `/logs` for errors or click on the `/admin/backups.json` link in the error page to get additional details about the error.
So I couldn’t find anything interesting in logs on the host but i did look at the Discourse logs and noticed that it reports a slightly different error to the `/admin/bakups.json`
```plaintext
Failed to list backups from S3: The difference between the request time and the current time is too large.
```
Notice the extra prefix: **Failed to list backups from S3**
---
### Author:  [@gerhard](https://meta.discourse.org/u/gerhard)
#### Post date: [February 14, 2019, 11:07am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/8 "2019-02-14T11:07:30Z")
I’ve seen a lot of different errors from S3, but this one is new. Is your server using a timezone other than UTC? If so, try changing it to UTC.
---
### Author:  [@modius](https://meta.discourse.org/u/modius)
#### Post date: [February 14, 2019, 11:19am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/9 "2019-02-14T11:19:50Z")
> [@gerhard](#):
>
> I’ve seen a lot of different errors from S3, but this one is new. Is your server using a timeline other than UTC? If so, try changing it to UTC.
Well I’ve tried resetting the NTP to point at AWS servers.. and just about everything else here:
>
Changed the AWS access key… it’s starting to get my goat.
---
### Author:  [@modius](https://meta.discourse.org/u/modius)
#### Post date: [February 14, 2019, 11:30am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/10 "2019-02-14T11:30:52Z")
> [@modius](#):
>
> So that was promising… but backup view still dead with the same error. So then I restarted the app:
Argh.
Was dealing with two Discourse forums with similar IPs. Updating the time on one and checking the backups on the other. Looks like both are working now! Huzzah 🙌
AWS has an issue once the server falls out of sync by 5 minutes or more:
Fix for Ubuntu was to reset the clock.
```plaintext
$ date
Thu Feb 14 05:32:16 EST 2019
$ ntpdate ntp.ubuntu.com
14 Feb 05:26:58 ntpdate[12447]: step time server 91.189.91.157 offset -408.660323 sec
$ date
Thu Feb 14 05:27:02 EST 2019
```
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [March 16, 2019, 11:38am UTC](https://meta.discourse.org/t/backups-have-started-failing-due-to-server-time-being-wrong/108802/11 "2019-03-16T11:38:51Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.