I have a handful of Discourse instances that ran for a very long tome without issues. Recently I moved to Hetzner from DO. As part of the move I finally initiated monitoring, using Uptime for now. I hardened the servers using Ubuntu USG.
Three of the 7 instances have routinely going offline for 5 to 15 minutes around the same time daily. Perhaps this was occurring on DO but without monitoring who knows.
All site are running the same cast of Official plugins.
Suspecting a memory issue I upgraded one instance from 2 CPU 2GB RAM to 3 CPU with 4GB RAM
Did not change db_shared_buffers: “256MB”
Did not change UNICORN_WORKERS: 4
The increased performance seems to have helps but should I need so much resources on a very quiet site?
Any thoughts or suggestions would be appreciated.
The incident sequence always starts with a Job exception error
Message (4 copies reported)
Job exception: Waited 1.0 seconds
Backtrace
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/buffered_io.rb:214:in `block in fill_buffer'
<internal:kernel>:187:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/buffered_io.rb:197:in `fill_buffer'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/buffered_io.rb:187:in `ensure_remaining'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/buffered_io.rb:152:in `getbyte'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/ruby_connection.rb:96:in `read'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/connection_mixin.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client.rb:316:in `block (2 levels) in call_v'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client/middlewares.rb:16:in `call'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client.rb:315:in `block in call_v'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client.rb:721:in `ensure_connected'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-client-0.25.0/lib/redis_client.rb:314:in `call_v'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-5.4.0/lib/redis/client.rb:90:in `call_v'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/rack-mini-profiler-4.0.0/lib/mini_profiler/profiling_methods.rb:90:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-5.4.0/lib/redis.rb:152:in `block in send_command'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-5.4.0/lib/redis.rb:151:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-5.4.0/lib/redis.rb:151:in `send_command'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/redis-5.4.0/lib/redis/commands/strings.rb:191:in `get'
/var/www/discourse/lib/discourse_redis.rb:144:in `public_send'
/var/www/discourse/lib/discourse_redis.rb:144:in `block (3 levels) in <class:DiscourseRedis>'
/var/www/discourse/lib/discourse_redis.rb:29:in `ignore_readonly'
/var/www/discourse/lib/discourse_redis.rb:144:in `block (2 levels) in <class:DiscourseRedis>'
/var/www/discourse/lib/sidekiq/pausable.rb:19:in `paused?'
/var/www/discourse/lib/sidekiq/pausable.rb:105:in `paused?'
/var/www/discourse/config/initializers/100-sidekiq.rb:98:in `block (3 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/mini_scheduler-0.18.0/lib/mini_scheduler.rb:74:in `block (2 levels) in start'
Followed soon after with something like:
Failed to warm up pretty text: terminated /var/www/discourse/lib/discourse_js_processor.rb:89:in `call' /var/www/discourse/lib/discourse_js_processor.rb:89:in `block in v8_call' /var/www/discourse/lib
Message (2 copies reported)
Failed to warm up pretty text: terminated
/var/www/discourse/lib/discourse_js_processor.rb:89:in `call'
/var/www/discourse/lib/discourse_js_processor.rb:89:in `block in v8_call'
/var/www/discourse/lib/discourse_js_processor.rb:88:in `synchronize'
/var/www/discourse/lib/discourse_js_processor.rb:88:in `v8_call'
/var/www/discourse/lib/discourse_js_processor.rb:113:in `perform'
/var/www/discourse/lib/pretty_text.rb:34:in `apply_es6_file'
/var/www/discourse/lib/pretty_text.rb:41:in `block in ctx_load_directory'
/var/www/discourse/lib/pretty_text.rb:39:in `each'
/var/www/discourse/lib/pretty_text.rb:39:in `ctx_load_directory'
/var/www/discourse/lib/pretty_text.rb:81:in `create_es6_context'
/var/www/discourse/lib/pretty_text.rb:135:in `block in v8'
/var/www/discourse/lib/pretty_text.rb:133:in `synchronize'
/var/www/discourse/lib/pretty_text.rb:133:in `v8'
/var/www/discourse/lib/pretty_text.rb:176:in `block in markdown'
/var/www/discourse/lib/pretty_text.rb:678:in `block in protect'
/var/www/discourse/lib/pretty_text.rb:678:in `synchronize'
/var/www/discourse/lib/pretty_text.rb:678:in `protect'
/var/www/discourse/lib/pretty_text.rb:175:in `markdown'
/var/www/discourse/lib/pretty_text.rb:301:in `cook'
/var/www/discourse/lib/discourse.rb:960:in `after_fork'
config/unicorn.conf.rb:197:in `block in reload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:684:in `init_worker_process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:721:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:561:in `maintain_worker_count'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:294:in `join'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/w...
Backtrace
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:134:in `block in error'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `block in dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `each'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:231:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/activesupport-7.2.2.1/lib/active_support/broadcast_logger.rb:134:in `error'
/var/www/discourse/lib/discourse.rb:962:in `rescue in after_fork'
/var/www/discourse/lib/discourse.rb:958:in `after_fork'
config/unicorn.conf.rb:197:in `block in reload'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:684:in `init_worker_process'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:721:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:561:in `maintain_worker_count'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:294:in `join'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.3.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.3.0/bin/unicorn:25:in `<main>'
`
``