# Sidekiq runit script too fragile
**URL:** https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331
**Category:** Support
**Tags:** sidekiq
**Created:** [November 21, 2025, 11:56am UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331 "2025-11-21T11:56:44Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@hel\_Sinki](https://meta.discourse.org/u/hel_Sinki)
#### Post date: [November 21, 2025, 11:56am UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/1 "2025-11-21T11:56:44Z")
Hi team,
reporting a failure mode in the official Docker/runit setup that can silently kill Sidekiq (and therefore AI / background jobs) without any rebuild or upgrade.
### **Environment**
- Official Discourse Docker install (standard container + runit services).
- No rebuild/upgrade right before the issue started.
- Discourse AI plugin enabled, but AI stopped replying.
### **Symptoms**
- AI looks enabled in admin UI, but no AI replies appear.
- Background jobs (AI/embeddings/auto-reply) appear stuck.
- sv status sidekiq shows Sidekiq repeatedly dying right after start:
```plaintext
down: sidekiq: 1s, normally up, want up
```
- Manually starting Sidekiq works fine, so the app itself is OK:
```plaintext
bundle exec sidekiq -C config/sidekiq.yml
# stays up, connects to Redis, processes jobs
```
### **What we found**
The default runit script was:
```plaintext
exec chpst -u discourse:www-data \
bash -lc 'cd /var/www/discourse && ... bundle exec sidekiq -e production -L log/sidekiq.log'
```
Two fragility points:
1. **Primary group** **www-data** In my container, typical writable paths are owned by discourse:discourse. Any drift in tmp/pids or shared paths can make Sidekiq exit during boot when run under www-data, even though manual start as discourse works.
2. **Forced** **-L log/sidekiq.log** **writing to shared logs** The log path is a symlink into /shared/log/rails/sidekiq.log. If that file/dir gets recreated with different ownership/permissions, Sidekiq can exit immediately before producing useful logs.
### **Related trigger: logrotate failing daily**
Separately, logrotate was failing every day with:
```plaintext
error: skipping "...log" because parent directory has insecure permissions
Set "su" directive in config file ...
```
Cause was standard Debian/Ubuntu perms:
- /var/log is root:adm with 0775 (group writable).
- logrotate refuses rotation unless a global su directive is set.This is expected upstream behavior.
At the moment the daily logrotate job failed, it also recreated files under /shared/log/rails/ (including sidekiq.log), which likely interacted with the forced -L logging and contributed to the Sidekiq “1s crash” loop.
### **Fix (no rebuild needed)**
1. **Fix logrotate so it stops touching shared logs in a failed state** Add a global su directive:
```plaintext
# /etc/logrotate.conf (top)
su root adm
```
After that, logrotate -v exits 0 and no longer reports insecure parent perms.
1. **Replace Sidekiq runit script with a more robust default** Switching to discourse:discourse and the standard sidekiq.yml, and **not forcing** **-L log/sidekiq.log** , makes Sidekiq stable:
```plaintext
#!/bin/bash
exec 2>&1
cd /var/www/discourse
mkdir -p tmp/pids
chown discourse:discourse tmp/pids || true
exec chpst -u discourse:discourse \
bash -lc 'cd /var/www/discourse && rm -f tmp/pids/sidekiq*.pid; exec bundle exec sidekiq -C config/sidekiq.yml'
```
After this:
- sv status sidekiq stays run:
- AI/background jobs resume.
### **Request / suggestion**
Could we consider making the official Docker/runit Sidekiq service more robust by default?
For example:
- Run Sidekiq under discourse:discourse (matching typical ownership inside container).
- Prefer bundle exec sidekiq -C config/sidekiq.yml.
- Avoid forcing a shared log file via -L log/sidekiq.log, or make it resilient to logrotate/shared-volume perms drift.
Even a doc note (“if Sidekiq shows down: 1s but manual start works, check /etc/service/sidekiq/run and avoid forced shared logging”) would help self-hosters a lot.
Happy to provide more logs if needed. Thanks!
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [November 22, 2025, 10:55pm UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/2 "2025-11-22T22:55:46Z")
> [@hel\_Sinki](#):
>
> The default runit script was:
>
> ```plaintext
> exec chpst -u discourse:www-data \
> bash -lc 'cd /var/www/discourse && ... bundle exec sidekiq -e production -L log/sidekiq.log'
>
> ```
Where are you finding that? Sidekiq is launched via the unicorn master to conserve memory. Not seeing this code at all in discourse\_docker. Looks like maybe you are using a very old setup?
---
### Author:  [@hel\_Sinki](https://meta.discourse.org/u/hel_Sinki)
#### Post date: [November 23, 2025, 2:30pm UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/3 "2025-11-23T14:30:02Z")
Hi — let me restate this strictly based on runtime facts from the official Docker container.
### **What I’m seeing in the running container (facts)**
This is an **official Docker install with runit** (standard /var/discourse launcher workflow; no rebuild right before the incident). Inside the container:
1. **A runit Sidekiq service exists and is the one being supervised**
```plaintext
ls -l /etc/service/sidekiq/run
sv status sidekiq
```
Output during the incident:
```plaintext
down: sidekiq: 1s, normally up, want up
```
1. **Manual Sidekiq start works**
```plaintext
cd /var/www/discourse
sudo -u discourse bundle exec sidekiq -C config/sidekiq.yml
```
This stays up, connects to Redis, and processes jobs.
1. **Patching only** **/etc/service/sidekiq/run** **(no rebuild) fixes the crash loop immediately** Replaced /etc/service/sidekiq/run with:
```plaintext
#!/bin/bash
exec 2>&1
cd /var/www/discourse
mkdir -p tmp/pids
chown discourse:discourse tmp/pids || true
exec chpst -u discourse:discourse \
bash -lc 'cd /var/www/discourse && rm -f tmp/pids/sidekiq*.pid; exec bundle exec sidekiq -C config/sidekiq.yml'
```
After that:
```plaintext
sv status sidekiq
run: sidekiq: (pid ) s
```
So Sidekiq is **not being launched via Unicorn master in this image** ; it’s a runit service whose runtime script can crash-loop.
### **Why you may not see the exact code in**
### **discourse\_docker**
I agree the literal text may not be in the repo because **/etc/service/sidekiq/run is a runtime artifact generated/injected during image build/boot** , not necessarily a verbatim file in discourse\_docker. But it _is_ the active supervised service in this official image, as shown above.
### **What triggered the fragility (facts + minimal inference)**
- We also observed daily logrotate failures due to standard Debian perms:/var/log = root:adm 0775, so logrotate refused rotation until adding global su root adm.
- When logrotate was failing, it recreated files under /shared/log/rails/, including sidekiq.log.
- The default runit script in this image used discourse:www-data and forced -L log/sidekiq.log into /shared/log, which makes Sidekiq **very sensitive to shared-volume perms drift** and can cause an immediate exit before useful logs.
### **Request / proposal**
Given the above, could we consider hardening the default Docker/runit Sidekiq service?
Suggested defaults:
- run as discourse:discourse (matches typical ownership inside container),
- start via bundle exec sidekiq -C config/sidekiq.yml,
- avoid forcing a shared -L log/sidekiq.log (or make it resilient).
This would prevent the silent down: 1s crash loop that stops all background/AI jobs.
Happy to test any branch/commit you point me at.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [November 23, 2025, 10:43pm UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/4 "2025-11-23T22:43:15Z")
Again … I am confused about where you are getting your image from:
This is the official image.
This is a search for the word sidekiq in the official discourse docker.
[https://github.com/search?q=repo%3Adiscourse%2Fdiscourse\_docker%20sidekiq&type=code](https://github.com/search?q=repo%3Adiscourse%2Fdiscourse_docker%20sidekiq&type=code)
There are 3 hits… nothing about a runit unit. It is managed via unicorn.
---
### Author:  [@hel\_Sinki](https://meta.discourse.org/u/hel_Sinki)
#### Post date: [November 24, 2025, 5:17am UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/5 "2025-11-24T05:17:01Z")
Hi — thanks, that screenshot helps clarify the layout.
I agree that in the current official image Sidekiq is **not a separate runit service** (no /etc/service/sidekiq/). It is launched from the **unicorn runit service’s startup chain** , which matches your /etc/service listing.
My report is still about a **runtime failure mode of that Sidekiq launch path** , regardless of whether it lives in a standalone runit unit or inside unicorn/run:
**Facts from runtime on my VPS (official Docker, no rebuild/upgrade right before incident):**
1. Background jobs stopped and AI replies stopped.
2. Sidekiq entered an immediate crash loop (down: 1s) when started by the container’s supervisor/startup chain.
3. Manual start as discourse via bundle exec sidekiq -C config/sidekiq.yml stayed up and processed jobs, so the app/redis were fine.
4. At the same time, logrotate failures caused /shared/log/rails/sidekiq.log (and related paths) to be recreated with different perms; after stabilizing the Sidekiq launch command (run as discourse:discourse, use sidekiq.yml, **avoid forcing shared** **-L sidekiq.log** ), the crash loop stopped immediately.
> So the literal file /etc/service/sidekiq/run may not exist in this image — agreed — but the **Sidekiq launch step embedded in the unicorn runit service is brittle to shared-volume perms/logrotate drift** and can silently kill Sidekiq without a rebuild. That’s the core issue.
> **Suggestion:** please consider hardening the Sidekiq launch in the official unicorn runit script (or where it is generated):
- run Sidekiq under discourse:discourse,
- prefer bundle exec sidekiq -C config/sidekiq.yml,
- avoid forcing a shared -L log/sidekiq.log (or make it resilient).
---
### Author:  [@tobiaseigen](https://meta.discourse.org/u/tobiaseigen)
#### Post date: [December 3, 2025, 2:47am UTC](https://meta.discourse.org/t/sidekiq-runit-script-too-fragile/389331/6 "2025-12-03T02:47:03Z")