# Help! Discourse workers dying on our community site
**URL:** https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518
**Category:** Self-hosting
**Created:** [2017 年9 月 21 日 15:04 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518 "2017-09-21T15:04:44Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@digitaldan](https://meta.discourse.org/u/digitaldan)
#### Post date: [2017 年9 月 21 日 15:04 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/1 "2017-09-21T15:04:44Z")
HI, I’m a member of the [openHAB](http://openhab.org) team, we are one of the largest open source home automation projects. We are also affiliated with the [Eclipse Smart Home](http://www.eclipse.org/smarthome/) project which we contribute to, and use as the base of our own project (think of us as the reference implementation) .
Our community forums have been **down for the last 24 hours** and we are struggling to get them back online.
**The Issue**
Unicorn processes seem to be killed after 30 seconds, we see this in shared/standalone/log/rails/unicorn.stderr.log
```plaintext
E, [2017-09-21T14:29:18.466973 #261] ERROR -- : worker=0 PID:13791 timeout (43s > 30s), killing
E, [2017-09-21T14:29:18.467612 #261] ERROR -- : worker=2 PID:13823 timeout (36s > 30s), killing
E, [2017-09-21T14:29:18.467874 #261] ERROR -- : worker=1 PID:14395 timeout (33s > 30s), killing
E, [2017-09-21T14:29:18.547036 #261] ERROR -- : worker=0 PID:13791 timeout (43s > 30s), killing
E, [2017-09-21T14:29:18.547289 #261] ERROR -- : worker=1 PID:14395 timeout (33s > 30s), killing
```
Once this starts happening, we have to restart the container or this happens indefinitely. Most times it requires more then one restart to be functional again.
We **doubled** the size of our hardware 2 days ago, and now can not keep the site up for more then a few mins,
**What we have done so far**
For the last couple weeks this has been happening more frequently, and we have needed to restart it to bring it back online.
We were using a Digital Ocean 2 cpu / 4gb instance and made the decision to double that hoping that more mem and cpu will help handle the load.
After upgrading to 4 cpu / 8gb memory and rebuilding our container with an appropriate app.yml our problem seems to be worse now, not better. We still have the exact same errors, but now our site can’t seem to be up for more then a few minutes at a time. Our current settings look like the following
```plaintext
db_shared_buffers: "2GB"
db_work_mem: "40MB"
UNICORN_WORKERS: 4
```
I have also tried increasing the shared buffers, workers and a combination of both together, but nothing has seemed to help. Restarting no longer help as well.
Git log shows we are running revision `ee9a607677106603ace3e90f3c5bd9ca84a11ef7` which was commited on `Fri Sep 15 08:27:27 2017 +0800`
**Next Steps**
We are currently out of ideas after doubling the hardware. I’m not sure why the unicorn processes are hanging and then being killed under load, and why doubling the hardware has made things worse, not better.
any help by the community would be very, very appreciated.
Thanks
Dan-
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [2017 年9 月 21 日 16:21 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/2 "2017-09-21T16:21:25Z")
Any plugins installed?
---
### Author:  [@digitaldan](https://meta.discourse.org/u/digitaldan)
#### Post date: [2017 年9 月 21 日 16:50 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/3 "2017-09-21T16:50:27Z")
> [@pfaffman](#):
>
> Any plugins installed?
I can’t log in to the web interface, but this is what is listed in the plugin dir.
```plaintext
/var/www/discourse/plugins# ls
discourse-details discourse-nginx-performance-report discourse-solved lazyYT
discourse-github-linkback discourse-plugin-code-fences-buttons discourse-tagging plugins
discourse-narrative-bot discourse-presence
```
---
### Author:  [@digitaldan](https://meta.discourse.org/u/digitaldan)
#### Post date: [2017 年9 月 21 日 16:56 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/4 "2017-09-21T16:56:48Z")
> [@digitaldan](#):
>
> I can’t log in to the web interface, but this is what is listed in the plugin dir.
Sorry, still new to discourse admin, and here is what is in our app.yml
```plaintext
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-tagging.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/ThomDietrich/discourse-plugin-code-fences-buttons.git
- git clone https://github.com/discourse/discourse-github-linkback.git
```
---
### Author:  [@digitaldan](https://meta.discourse.org/u/digitaldan)
#### Post date: [2017 年9 月 21 日 18:14 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/5 "2017-09-21T18:14:47Z")
We have disabled all plugins, but our problem still remains.
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [2017 年9 月 21 日 18:57 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/6 "2017-09-21T18:57:39Z")
@sam saw this recently [with corrupt databases](https://meta.discourse.org/t/how-to-recover-from-a-very-slow-database/68793).
---
### Author:  [@digitaldan](https://meta.discourse.org/u/digitaldan)
#### Post date: [2017 年9 月 21 日 20:53 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/7 "2017-09-21T20:53:17Z")
All thank you for your responses! Just an FYI I was in the app looking at the DB (thanks @codinghorror for the link) , so I had done a `sv stop unicorn` and when I was done I did a `sv start unicorn`. When the unicorn server started back up our community site came back and now does not have any issues, and is very snappy to boot ! I had run `vacuum full` yesterday on the DB and it did not seem to help (or do anything, which didn’t surprise me as auto vacuum is on), but I did run that again between the start and stop. At this point I’m not sure what has happened to make it come back.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [2017 年9 月 26 日 18:16 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/8 "2017-09-26T18:16:29Z")
The system wasn’t exhibiting the signs that @sam described.
`du postgres_data` shows 9GB, so maybe the database is just too big? But it was working fine on a 4GB droplet before this problem started.
Load average is staying below 2.
Maybe it’s bad luck and that server at Digital Ocean is broken in some way.
@digitaldan, you might file a ticket with them and see if they know anything like the SSD is failing or something. Or some other user on that machine is breaking it.
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [2019 年10 月 22 日 07:43 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/9 "2019-10-22T07:43:24Z")
---
### Author:  [@system](https://meta.discourse.org/u/system)
#### Post date: [2023 年7 月 1 日 21:58 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/10 "2023-07-01T21:58:01Z")
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [2023 年7 月 1 日 22:00 UTC](https://meta.discourse.org/t/help-discourse-workers-dying-on-our-community-site/70518/11 "2023-07-01T22:00:42Z")