The discussion revolves around a Discourse forum experiencing issues with delayed updates, specifically the ‘Updating Last Seen’ and ‘hijack topics timings’ processes, which are still running after 90 seconds. The user, Dmitry_Krasnoperov, has a large database with many images, and is running on a 4CPU/8GB server with 2GB swap enabled.
Pfaffman inquires about the server’s RAM and CPU, and suggests checking the value of db_shared_buffers in the app.yml file. Dmitry_Krasnoperov shares the value, which is set to 2048MB, and later increases it to 3048MB.
Despite these changes, the issue persists, and Dmitry_Krasnoperov continues to experience disk activity peaks.
Wandagoner and Crius share similar experiences with the same error, suggesting that it may be related to the size of the database. Crius also mentions that the hosted Discourse may have customers with large databases, and shares their own database size, which is approximately 89G.
Several time a day my forum goes in situation where all updates process are delayed.
'Updating Last Seen' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'
'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'
'Track Visit' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/plugins/discourse-topic-previews/lib/topic_list_previews_helper.rb:13:in `create_topic_thumbnails'
/var/www/discourse/plugins/discourse-topic-previews/lib/cooked_post_processor_edits.rb:76:in `update_post_image'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:40:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:10:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:35:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:24:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb:185:in `execute_job'
Job exception: ERROR: deadlock detected
DETAIL: Process 22512 waits for ExclusiveLock on tuple (22418,9) of relation 16742 of database 16400; blocked by process 28074.
Process 28074 waits for ShareLock on transaction 1013165180; blocked by process 24391.
Process 24391 waits for ShareLock on transaction 1013163724; blocked by process 1150.
Process 1150 waits for ShareLock on transaction 1013156883; blocked by process 22512.
HINT: See server log for query details.
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:118:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:90:in `exec'
(eval):6:in `exec'
/var/www/discourse/app/models/post.rb:646:in `block in calculate_avg_time'
/var/www/discourse/lib/freedom_patches/active_record_base.rb:45:in `retry_lock_error'
/var/www/discourse/app/models/post.rb:623:in `calculate_avg_time'
/var/www/discourse/app/jobs/scheduled/calculate_avg_time.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/app/jobs/base.rb:277:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'
## Set db_shared_buffers to a max of 25% of the total memory.
##
## On 1GB installs set to 128MB (to leave room for other processes)
## on a 4GB instance you may raise to 1GB
db_shared_buffers: "2048MB"
