# Error "Killed" while running rake posts:rebake
**URL:** https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943
**Category:** Self-hosting
**Tags:** server-resources
**Created:** [March 18, 2019, 4:53pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943 "2019-03-18T16:53:37Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [March 18, 2019, 4:53pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/1 "2019-03-18T16:53:37Z")
> [@Rebake all posts?](https://meta.discourse.org/t/rebake-all-posts/69340/2):
>
> rake posts:rebake
On a new Ubuntu 18 host, I’m having a problem with this step:
```plaintext
# cd /var/discourse/
# ./launcher enter app
# rake posts:rebake
Rebaking post markdown for 'default'
5 / 2950 ( 0.2%)/usr/local/bin/rake: line 2: 6788 Killed RAILS_ENV=production sudo -H -E -u discourse bundle exec bin/rake "$@"
# echo $?
137
```
In troubleshooting, I did a `./launcher rebuild all` and then ran the rebake command, I got a bit further:
```plaintext
# rake posts:rebake
Rebaking post markdown for 'default'
503 / 2950 ( 17.1%)/usr/local/bin/rake: line 2: 4588 Killed RAILS_ENV=production sudo -H -E -u discourse bundle exec bin/rake "$@"
# echo $?
137
```
what would cause the rebuild to fail (lack of memory? but this is a 2GB host, and only 2600 posts.)
EDIT - I’ve remove everything about missing image.. and re-asked that [here.](https://meta.discourse.org/t/images-lost-after-restore-from-backup/112289)
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [March 18, 2019, 6:03pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/2 "2019-03-18T18:03:15Z")
You shouldn’t need to rebake if you kept using the same domain.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [March 18, 2019, 7:41pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/3 "2019-03-18T19:41:53Z")
Ok, so now I have two problems 😂
I can’t rebake, i do have a lot of (all?) images showing as broken.
I am 99.9% certain that the backups were set to include images…
I see files in `shared/standalone/uploads/default/original/1X/` (etc\_ but none in `default/238/*`
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [March 18, 2019, 7:51pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/4 "2019-03-18T19:51:00Z")
Do you still have both servers up? Can you compare folders? Maybe `rsync` between old and new and see the differences?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [March 18, 2019, 8:50pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/5 "2019-03-18T20:50:18Z")
The old server is gone (the process seemed to go so smoothly, I didn’t think to check images before shutting the old one down)
Would rebaking re-create the missing images?
Any thoughts as to why it’s failing?
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [March 19, 2019, 6:44am UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/6 "2019-03-19T06:44:34Z")
This evening I rebooted the host (though it’s uptime was under 2 weeks)
rebaking succeeded
```plaintext
# rake posts:rebake
Rebaking post markdown for 'default'
2950 / 2950 (100.0%)
2950 posts done!
-------------------------
```
but, as you suggested, this didn’t bring back the images. Ill start a new thread for that.
As for this issue (not being able to rebake ~2600 posts) Ill note that this thread seems useful:
> [@My journey into a massive posts rebake job](https://meta.discourse.org/t/my-journey-into-a-massive-posts-rebake-job/84816/23):
>
> If you need to rebake all posts instantly then run bundle exec rake posts:rebake. Post.update\_all("baked\_version = NULL") will rebake 100 posts (by default) every 15 minutes.
EDIT: As for the initial issueI was trying to solve, this thread indicates that `recover_from_tombstone` is needed in addition to the rebuild, but that hasn’t worked yet.
> [@Old image uploads become broken images](https://meta.discourse.org/t/old-image-uploads-become-broken-images/47236/2):
>
> Lest anyone thinks that this query went unanswered, many thanks to @zogstrip for solving this. It seems that Discourse got confused about whether those uploaded images were being used and moved them for subsequent deletion. Though you will probably never need it, the fix went something like this: # copy the "deleted" images from their to-be-deleted staging area cd /var/discourse ./launcher enter app rake uploads:recover\_from\_tombstone # rebake to see that it doesn't happen again rake posts:re…
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [March 20, 2019, 2:40pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/9 "2019-03-20T14:40:08Z")
I tried `recover_from_tombstone`. The command exited without warnings, and so I’m trying to rebuild.
That process fails at random places. I’ve tried rebooting the host, rebuilding the app, and it fails anywhere from 5-55% of the way through the rebake process. Note that there are only ~2950 posts.. pretty small by discourse’s standards.
I am running a number of plugins, I’m not sure if that’s related:
```
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/cpradio/discourse-plugin-checklist.git
- git clone https://github.com/discourse/discourse-assign.git
- git clone https://github.com/discourse/discourse-data-explorer.git
- git clone https://github.com/discourse/discourse-feature-voting.git
- git clone https://github.com/discourse/discourse-push-notifications.git
- git clone https://github.com/discourse/discourse-chat-integration.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/discourse/discourse-staff-notes.git
- git clone https://github.com/watchmanmonitoring/discourse-saml.git # cloned from discourse, to allow for more settings.
```
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 10, 2019, 8:58pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/10 "2019-05-10T20:58:50Z")
I’ve tried a couple of times, and still had errors.
Since I posted, the missing images were fixed, and I’ve rebuilt the container. The rebake still failed, but after a reboot of the host I was able to get the rebake to succeed.
```plaintext
# ./launcher enter app
root@community-2019-app:/var/www/discourse# rake posts:rebake
Rebaking post markdown for 'default'
2976 / 2976 (100.0%)
2976 posts done!
```
---
### Author:  [@supermathie](https://meta.discourse.org/u/supermathie)
#### Post date: [May 10, 2019, 9:11pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/11 "2019-05-10T21:11:37Z")
> [@watchmanmonitor](#):
>
> Killed
FYI this nearly always means “ran out of memory or some other hard limit”
You can _probably_ see a log entry in your kernel ring buffer (`dmesg`) saying something got killed to free memory.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [May 11, 2019, 12:39am UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/12 "2019-05-11T00:39:14Z")
You mentioned it’s a 2GB server, but does it have swap?
> [@Create a swapfile for your Linux server](https://meta.discourse.org/t/create-a-swapfile-for-your-linux-server/13880):
>
> bulb For servers with \<= 4GB of RAM, running ./discourse-setup will prompt for and automatically create a 2GB swapfile. Most cloud virtual machine providers do not set up swapfiles as part of their server provisioning. In particular, upgrading Discourse produces a lot of memory pressure. With a swap file, rather than randomly terminating processes with an out of memory error, things will slow down instead. Having a swap file is a cheap insurance policy that protects you from many other lo…
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 5:23pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/13 "2019-05-12T17:23:01Z")
Now that you mention it, no.
```plaintext
root@community-2019:~# swapon --show
root@community-2019:~# free -h
total used free shared buff/cache available
Mem: 1.9G 1.4G 193M 70M 317M 273M
Swap: 0B 0B 0B
root@community-2019:~#
```
I followed the steps here at this version of [INSTALL-cloud.md](https://github.com/discourse/discourse/blob/444bc466b06e9490d763652ece4d92a5a5379973/docs/INSTALL-cloud.md)
Which don’t mention swap, and so I didn’t even consider it, given that I wasn’t using the bare minimum 1GB.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 5:30pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/14 "2019-05-12T17:30:19Z")
> [@Stephen](#):
>
> You mentioned it’s a 2GB server, but does it have swap?
I’ve added swap via [Digital Ocean’s guide](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-18-04). Will see how this goes.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [May 12, 2019, 5:31pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/15 "2019-05-12T17:31:45Z")
That’s it, without a doubt. On 1GB it’s mandatory, on anything more than 1GB it’s only a matter of time before processes are killed when the system runs low on free memory:
> [@Worker process is more than defined?](https://meta.discourse.org/t/worker-process-is-more-than-defined/113230/4):
>
> Our recommended configuration is at least 2G of RAM + 2G of swap.
> [@schleifer](#):
>
> Yeah ⬆ that. The linux kernel effectively _demands_ swap be available or OOM killer will wreck your 💩 sooner or later.
The way linux handles memory changes fundamentally if you don’t have swap available. Think of an Android smartphone terminating background processes for hungrier foreground processes.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 9:51pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/16 "2019-05-12T21:51:21Z")
Sure, makes sense.
I just did a rebake again, and sure enough, it was eating in to the new swap.
```ruby
CPU[||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 77, 84 thr; 1 running
Mem[|||||||||||||||||||||||||||||||||||||||1.58G/1.95G] Load average: 2.05 1.51 1.15
Swp[|||||||||||||||||||||| 431M/1024M] Uptime: 2 days, 01:58:25
```
and the rebake finished fine. 🙌
I don’t know if the powers-that-be want to (re)add a note about adding swap in the basic setup guide, but it would have prevented the issue here.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [May 12, 2019, 10:04pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/17 "2019-05-12T22:04:15Z")
When you run `discourse-setup`, it says:
> WARNING: Discourse requires at least 2GB of swap when running with 2GB of RAM
> or less. This system does not appear to have sufficient swap space.
>
> Without sufficient swap space, your site may not work properly, and future
> upgrades of Discourse may not complete successfully.
>
> Ctrl+C to exit or wait 5 seconds to have a 2GB swapfile created.
I guess the assumption is that power users who bypass `discourse-setup` already understand how to configure a server.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 10:11pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/18 "2019-05-12T22:11:11Z")
Sure but I had 2GB ram, and have nothing but discourse on this host, and still ran in to issues.
It’s quite possible that the `rebake` process is the _only_ thing that would cause me grief, and I’m not sure I would have been doing that if hadn’t hit some obscure issue on my host, when trying to restore from a backup.
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [May 12, 2019, 10:12pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/19 "2019-05-12T22:12:02Z")
Correct, because as I quoted
> [@Stephen](#):
>
> Discourse requires **at least 2GB of swap** when running **with 2GB of RAM**
That perfectly describes your situation.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 10:13pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/20 "2019-05-12T22:13:53Z")
oh, you think that maybe I skipped a step? Wouldn’t shock me.. I was setting up the new host knowing full well that I was just going to restore from backup. There’s a good chance that I never ran `discourse-setup`
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [May 12, 2019, 10:15pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/21 "2019-05-12T22:15:45Z")
If you copied the `app.yml` across you skipped that step entirely you’re basically in ‘no seatbelt’ mode.
You wouldn’t see the message about that, or any other of the system requirement checks which can save you pain down the line.
`Discourse-setup` doesn’t just create a fresh app.yml, it checks all of the common gotchas which hinder new installs. Docker, disk, memory, scaling defaults, whether ports are already in use, it’s all there.
[Take a look](https://github.com/discourse/discourse_docker/blob/master/discourse-setup), it’s quite comprehensive.
What would be useful would be a way to pipe a migrated app.yml into discourse-setup, so that site-specific stuff could be treated as a default and preserved, while the other stuff would be configured for the system for the first time.
---
### Author:  [@watchmanmonitor](https://meta.discourse.org/u/watchmanmonitor)
#### Post date: [May 12, 2019, 10:19pm UTC](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943/22 "2019-05-12T22:19:38Z")
Thanks @Stephen.. yeah I was out in a wasteland because I was “just” restoring from a backup.
In the end, anyone else having issues with rebake can find this thread & its solution. Also, due to the work @vinothkannans & team did recently relating to migrating images, my specific issue should be a moot point going forward.
Thanks again for the pointer on swap, it’s the solution to the problem I started with this thread.
[Next page](https://meta.discourse.org/t/error-killed-while-running-rake-posts-rebake/111943.md?page=2)