Can't start development instance-- renice: failed to set priority

Development

AI-generated summary

Users are encountering a development instance startup failure due to a renice: failed to set priority error in Discourse’s Sidekiq worker initialization. pfaffman reports the error began unexpectedly despite no recent code changes, and notes the problematic line — Discourse::Utils.execute_command("renice", "-n", "5", "-p", Process.pid.to_s) — has been in place for four years. They temporarily bypassed the issue by commenting it out, but remain puzzled about the root cause, suspecting possible kernel or system policy changes. Ed_S suggests potential culprits like updated system utilities or UID/priority policy shifts, though clarifies they lack experience with local dev setups. pfaffman confirms this is a native Linux dev instance, not containerized. Later, andromedarabbit reports the same error occurring in a local Kubernetes environment, indicating the issue may not be isolated to one setup. The error consistently arises during Sidekiq worker forking, where renice fails with “Permission denied,” causing worker crashes and restarts. No definitive fix is offered, but the recurrence across environments suggests a broader system-level permission or configuration issue affecting process priority adjustments.

1

I don’t know what might have changed since the last time I cranked up this dev instance, but now it’s failing like this:

Sidekiq client with Sidekiq::RedisConnection::RedisAdapter options {:host=>"localhost", :port=>6379, :namespace=>"sidekiq"}
Sidekiq client with Sidekiq::RedisConnection::RedisAdapter options {:host=>"localhost", :port=>6379, :namespace=>"sidekiq"}
Sidekiq client with Sidekiq::RedisConnection::RedisAdapter options {:host=>"localhost", :port=>6379, :namespace=>"sidekiq"}
I, [2023-05-17T13:31:56.328911 #1103707]  INFO -- : worker=0 ready                                                           
I, [2023-05-17T13:31:56.330855 #1103710]  INFO -- : worker=1 ready                                                         
Loading Sidekiq in process id 1103706                                                                                        
Sidekiq client with Sidekiq::RedisConnection::RedisAdapter options {:host=>"localhost", :port=>6379, :namespace=>"sidekiq"}  
/home/pfaffman/src/discourse-repos/discourse/lib/discourse.rb:137:in `exec': renice: failed to set priority for 1103706 (proc
ess ID): Permission denied                                                                                                   
/home/pfaffman/src/discourse-repos/discourse/lib/discourse.rb:171:in `execute_command'   
/home/pfaffman/src/discourse-repos/discourse/lib/discourse.rb:137:in `exec'                                             
/home/pfaffman/src/discourse-repos/discourse/lib/discourse.rb:33:in `execute_command'                                
/home/pfaffman/src/discourse-repos/discourse/lib/demon/sidekiq.rb:51:in `after_fork'
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:155:in `block in run'                            
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:151:in `fork'
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:151:in `run'                                                  
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:146:in `start'                                   
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:14:in `block in start'                                        
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:14:in `times'                                               
/home/pfaffman/src/discourse-repos/discourse/lib/demon/base.rb:14:in `start'                                                 
/home/pfaffman/src/discourse-repos/discourse/config/unicorn.conf.rb:89:in `block in reload'                                  
/home/pfaffman/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:541:in `spawn_missing_workers'                                                                                                                 
/home/pfaffman/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/home/pfaffman/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/home/pfaffman/src/discourse-repos/discourse/bin/unicorn:96:in `load'                
/home/pfaffman/src/discourse-repos/discourse/bin/unicorn:96:in `block in <main>'    
/home/pfaffman/src/discourse-repos/discourse/bin/unicorn:95:in `fork'                                           
/home/pfaffman/src/discourse-repos/discourse/bin/unicorn:95:in `<main>'                                                      
I, [2023-05-17T13:31:56.356176 #1103713]  INFO -- : worker=2 ready                                                         
E, [2023-05-17T13:31:56.452446 #1103437] ERROR -- : reaped #<Process::Status: pid 1103706 exit 1> worker=unknown             
Detected dead worker 1103706, restarting...

And I did the only other thing I could think of: reboot. :shrimp: (the person shrugging emoji seems to be missing–and I just rebooted–I guess they’re trying to try out the new font awesome icons–so here’s a shrimp.)

2

I’m still confused on this one. I commented out this line in lib/demon/sidekiq.rb:

     #Discourse::Utils.execute_command("renice", "-n", "5", "-p", Process.pid.to_s)

That line’s been there for 4 years and it worked the last time I cranked this up. I don’t know what might have changed. Kernel upgrade, maybe? But that doesn’t make sense either.

3

An update to the docker image could have included an update to utilities such as renice. Or some kind of policy update within the container might have got in the way, if for example different UIDs are used for different processes in different ways. Or possibly the base priority has changed so the renice is now trying to increase priority, which it can’t do.

4

This is for a dev instance on Linux, not in the container. Seems strange that I’m the only one, but I’m often able to do silly things that no one else thinks of.

5

Ah, sorry - I have no experience with dev instances.

6

Same here. I found the same problem just a few hours ago on my local Kubernetes.

E, [2023-07-27T13:09:27.525053 #74] ERROR -- : reaped #<Process::Status: pid 456 exit 1> worker=unknown
Detected dead worker 456, restarting...
/var/www/discourse/lib/discourse.rb:137:in `exec': renice: failed to set priority for 483 (process ID): Permission denied
/var/www/discourse/lib/discourse.rb:171:in `execute_command'
/var/www/discourse/lib/discourse.rb:137:in `exec'
/var/www/discourse/lib/discourse.rb:33:in `execute_command'
/var/www/discourse/lib/demon/sidekiq.rb:51:in `after_fork'
/var/www/discourse/lib/demon/base.rb:155:in `block in run'
/var/www/discourse/lib/demon/base.rb:151:in `fork'
/var/www/discourse/lib/demon/base.rb:151:in `run'
/var/www/discourse/lib/demon/base.rb:146:in `start'
/var/www/discourse/lib/demon/base.rb:132:in `ensure_running'
/var/www/discourse/lib/demon/base.rb:31:in `block in ensure_running'
/var/www/discourse/lib/demon/base.rb:31:in `each'
/var/www/discourse/lib/demon/base.rb:31:in `ensure_running'
config/unicorn.conf.rb:243:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:295:in `join'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'