# Discourse failed to backup, how to debug?
**URL:** https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217
**Category:** Support
**Created:** [2016年十月29日 23:05 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217 "2016-10-29T23:05:28Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@trandatnh](https://meta.discourse.org/u/trandatnh)
#### Post date: [2016年十月29日 23:05 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/1 "2016-10-29T23:05:28Z")
Hi,
My discourse keeps failing backup for the last 2 days
Here is the backup logs:
> [2016-10-29 22:58:43] Making sure ‘/var/www/discourse/tmp/backups/default/2016-10-29-225843’ exists…
> [2016-10-29 22:58:43] Backup process was cancelled!
> [2016-10-29 22:58:43] Notifying ‘ltd’ of the end of the backup…
The directory `/var/www/discourse/tmp/backups/default/2016-10-29-225843` exists
At the parent directory, I found 5 empty directories. I guess 4 are created manually and 1 automatically created.
```plaintext
root@daynhauhoc-app:/var/www/discourse/tmp/backups/default# ls -lrth
total 20K
drwxr-xr-x 2 discourse www-data 4.0K Oct 29 07:17 2016-10-29-071717
drwxr-xr-x 2 discourse www-data 4.0K Oct 29 08:00 2016-10-29-080055
drwxr-xr-x 2 discourse www-data 4.0K Oct 29 08:14 2016-10-29-081441
drwxr-xr-x 2 discourse www-data 4.0K Oct 29 21:08 2016-10-29-210807
drwxr-xr-x 2 discourse www-data 4.0K Oct 29 22:58 2016-10-29-225843
```
* * *
I’m using 1.7.0.beta6, latest commit on 28 Oct 16.
[https://github.com/discourse/discourse/commit/4d58a00387f6c911b3d35c126289552f93044acc](https://github.com/discourse/discourse/commit/4d58a00387f6c911b3d35c126289552f93044acc)
I have done a successfully manual backup right before this issue.
Before that the automatically backup worked fine, but not now.
I got no ideas how to debug this issue.
Thanks
---
### Author:  [@Steven](https://meta.discourse.org/u/Steven)
#### Post date: [2016年十月30日 01:39 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/2 "2016-10-30T01:39:18Z")
I have some weird behavior too. The first time I launch a backup nothing happens, and when I cancel it’s on the “notifying \*\*\* of the end of the backup”.
But on another Discourse up to date ([i’m on this commit](https://github.com/discourse/discourse/commit/7953a53cc5fabcec231c35b3bbda4eef482930cf)), I didn’t have any issue during the backup (except I never received the notification)
I don’t think plugins can impact backups, but here are mine
```
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/iunctis/iunctis-toolbar.git
- git clone https://github.com/discourse/discourse-spoiler-alert.git
- git clone https://github.com/iunctis/vb_emoji.git
- git clone https://github.com/iunctis/discourse-affiliate.git
```
But I had no issue during my backup last night and I didn’t do any upgrade since then. Weird.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [2016年十月30日 15:15 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/3 "2016-10-30T15:15:41Z")
Can confirm manual backups are broken, and I think this only affects non-English boards, what’s the language of your boards?
---
### Author:  [@thess](https://meta.discourse.org/u/thess)
#### Post date: [2016年十月30日 17:14 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/4 "2016-10-30T17:14:37Z")
@trandatnh - I am having the same problem since updating to 1.7.0-beta6 (first upgrade since pre-beta). There seems to be no clear reasons in any logs indicating what may be failing as you have observed. I do notice that after the auto-backup starts, it appears to not complete. Cancelling the backup doesn’t really seem to be doing anything as attempting a manual backup just says “An operation is currently running. Can’t start a new job right now.”
@Falco - Our forum is English.
---
### Author:  [@Steven](https://meta.discourse.org/u/Steven)
#### Post date: [2016年十月30日 17:27 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/5 "2016-10-30T17:27:07Z")
French for me
I’ll try in english tonight, I’ll let you know
---
### Author:  [@trandatnh](https://meta.discourse.org/u/trandatnh)
#### Post date: [2016年十月31日 00:28 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/6 "2016-10-31T00:28:32Z")
@Falco It’s English board if you mean the default language. Although we discuss in Vietnamese
> **[Dạy Nhau Học](https://daynhauhoc.com)**
>
> Nơi để nhận giải đáp cho mọi câu hỏi, nơi để chia sẻ kiến thức, nơi để đọc, để suy ngẫm và để vui
This morning is worst, it automatically backup and failed to complete which forced the site to go to read-only mode. Cancel backup from the admin panel doesn’t work.
I’m running `./launcher rebuild app` and hope it will cease the backup for now.
**It works** , I disabled scheduling backup 😢 It’s bad to have no backup every day.
> [@thess](#):
>
> Cancelling the backup doesn’t really seem to be doing anything as attempting a manual backup just says “An operation is currently running. Can’t start a new job right now.”
Exactly, I have the same problem this morning as I mentioned above.
---
### Author:  [@Lapinot](https://meta.discourse.org/u/Lapinot)
#### Post date: [2016年十月31日 05:11 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/7 "2016-10-31T05:11:19Z")
Confirming, I can reproduce this failure (actually I’m struggling with backups too, see [here](https://meta.discourse.org/t/restore-on-stable-fail-on-mini-racer-0-1-3/52224)). Generally speaking, I find it quite questionable that the `tests-passed` branch is the default one in the yaml container file…
---
### Author:  [@leomakkinje](https://meta.discourse.org/u/leomakkinje)
#### Post date: [2016年十一月1日 09:54 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/8 "2016-11-01T09:54:19Z")
Same issue here. Since updating to “latest-release +95 1.7.0.beta6” there seems to be a stuck or broken backup process.
I first noticed it a day after the update when I received a notice about high CPU load on our forum VPS. Tried a reboot, didn’t fix. The top command lists a Ruby process that’s constantly consuming around 100% CPU.
Then I tried “cd /var/discourse; git pull; ./launcher rebuild app; ./launcher cleanup”. It initially failed because the database couldn’t be shutdown. A second attempt did work. The site was accessible again and no data was lost.
Then I tried to perform a manual backup. Normally I’d see a list of log messages but now all I see is a spinning wheel. Clicking Cancel gets the site out of ReadOnly mode for a few seconds and then it goes back to ReadOnly.
I’m going to do another rebuild and switch off automatic backups and see if that gets the site a) in permanent readwrite mode, b) keeps CPU load at a normal level. And then pray that a) the forum doesn’t crash, b) the bug gets fixed soon.
BTW, is there any way to roll back to a previous version, before 1.7.0.beta6?
---
### Author:  [@zogstrip](https://meta.discourse.org/u/zogstrip)
#### Post date: [2016年十一月1日 10:22 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/9 "2016-11-01T10:22:40Z")
We’re aware of the issues regarding backups and are actively working on fixing it. We have identified the problem but not the cause yet. It’s high on our list and will be fixed by the end of the week (hopefully sooner).
---
### Author:  [@ewanly](https://meta.discourse.org/u/ewanly)
#### Post date: [2016年十一月1日 12:20 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/10 "2016-11-01T12:20:03Z")
Mine is English but facing the same problem
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [2016年十一月1日 20:41 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/11 "2016-11-01T20:41:15Z")
Okay I’ve spent over a day on this now. No solution but have narrowed it down and I think @sam needs to look at it. I believe the bug is in mini\_racer, which seems to be crashing randomly during transpilation.
To reproduce the bug in development mode:
1. `rm -rf tmp`
2. `redis-cli flushall`
3. Create a backup in /admin/backups
It should crash on “Notifying ‘eviltrout’ of the end of the backup…”
The process will be using 100% CPU, and you _need to kill it_ before testing again.
Notes:
- The backup takes place in a fork from unicorn which is forked. I think this is important as running it from a rails console does not create the same issue. If you recall, we were able to crash discourse altogether when we were fiddling with PrettyText warming up before forking. I think miniracer is a little delicate when being forked in our app.
- The file it crashes while transpiling changes. Sometimes it’s the first file, sometimes it’s the fourth file, etc.
- Because some files will succeed, if you don’t `rm -rf tmp` the site will eventually start working as it will have cached all the files it needs to transpile. This is why it took forever to debug, because it would eventually fix itself!
- The mini racer is supposed to have a 15s timeout on eval, but even if you wait 15s it never continues.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2016年十一月2日 02:42 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/12 "2016-11-02T02:42:29Z")
Should be fixed per:
[https://github.com/discourse/discourse/commit/7e43e73df69a5ca70e7f4546465525c7392612fb](https://github.com/discourse/discourse/commit/7e43e73df69a5ca70e7f4546465525c7392612fb)
After we forked we correctly reset the `v8` context on pretty text, but the transpiler and js locale helper had `v8` context from parent.
Since v8 is not fork safe (and probably never will be) we must clear all our v8 context after forking.
Long term we should probably extend MiniRacer to allow it to “manually” free up all v8 contexts prior to forking and call a custom fork command, cause ideally prior to fork is the best way to do it. Also ruby really should give us a hook that we can call prior to forking.
Sadly, this has been on the back burner for so so long:
> **[Feature #5446: at\_fork callback API - Ruby - Ruby Issue Tracking System](https://bugs.ruby-lang.org/issues/5446)**
>
> Redmine
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [2016年十一月2日 06:22 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/13 "2016-11-02T06:22:38Z")
Sadly, not full sorted out … which is very odd, will continue to debug this
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [2016年十一月2日 14:26 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/14 "2016-11-02T14:26:11Z")
Update: Sam pushed another fix that does seem to fix this problem. I confirmed it is working this morning.
---
### Author:  [@ewanly](https://meta.discourse.org/u/ewanly)
#### Post date: [2016年十一月2日 16:05 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/15 "2016-11-02T16:05:21Z")
Yes, I just tried and its perfectly working 🙂 Thank you!
---
### Author:  [@eviltrout](https://meta.discourse.org/u/eviltrout)
#### Post date: [2016年十一月2日 16:11 UTC](https://meta.discourse.org/t/discourse-failed-to-backup-how-to-debug/52217/16 "2016-11-02T16:11:34Z")