'Updating Last Seen' and 'hijack topics timings' and 'Track Visit' is still running after 90 seconds

I’ve got the problem.

Several time a day my forum goes in situation where all updates process are delayed.

'Updating Last Seen' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'Track Visit' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/plugins/discourse-topic-previews/lib/topic_list_previews_helper.rb:13:in `create_topic_thumbnails'
/var/www/discourse/plugins/discourse-topic-previews/lib/cooked_post_processor_edits.rb:76:in `update_post_image'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:40:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:10:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:35:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:24:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb:185:in `execute_job'

I see stange disk activity as well

or some peek on week

I have backup settings as “every 3 days”.
I do have two plugins
discourse-whos-online
discourse-topic-list-previews

What’s to check? Or logs to see?

More errors:

Job exception: ERROR:  deadlock detected
DETAIL:  Process 22512 waits for ExclusiveLock on tuple (22418,9) of relation 16742 of database 16400; blocked by process 28074.
Process 28074 waits for ShareLock on transaction 1013165180; blocked by process 24391.
Process 24391 waits for ShareLock on transaction 1013163724; blocked by process 1150.
Process 1150 waits for ShareLock on transaction 1013156883; blocked by process 22512.
HINT:  See server log for query details.

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:118:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:90:in `exec'
(eval):6:in `exec'
/var/www/discourse/app/models/post.rb:646:in `block in calculate_avg_time'
/var/www/discourse/lib/freedom_patches/active_record_base.rb:45:in `retry_lock_error'
/var/www/discourse/app/models/post.rb:623:in `calculate_avg_time'
/var/www/discourse/app/jobs/scheduled/calculate_avg_time.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/app/jobs/base.rb:277:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'

Disable these and rebuild and see if that helps things.

1 Like

Do you have enough ram? I’d the cpu far enough?

How big is your database and how much ram do you have? Is swap enabled? Is it getting full?

4CPU/8GB and 2GB swap enabled.
But DB is large enough with a lot of images

   84.0GiB [##########] /uploads
   26.7GiB [###       ] /postgres_data
    4.7GiB [          ] /backups
    3.7GiB [          ] /log
  201.5MiB [          ] /redis_data
    1.7MiB [          ] /letsencrypt
  176.0KiB [          ] /postgres_run
   28.0KiB [          ] /state
   16.0KiB [          ] /tmp
   12.0KiB [          ] /ssl
e   4.0KiB [          ] /postgres_backup

I’ve just got this frozen status update and htop shows

and i got Error
'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!

backtrace

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'

Yep, will disable one by one and see.
Thanks

That’s a big database. What’s the value of db_shared_buffers in app.yml?

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ##
  ## On 1GB installs set to 128MB (to leave room for other processes)
  ## on a 4GB instance you may raise to 1GB
  db_shared_buffers: "2048MB"

You might try bumping it and the other memory variable up a bit. You have fast ssd?

Hi,

I’m in DigitalOcean SSD disk.
Set db_shared_buffers: “3048MB”
Disable plugins one by one.

but I’m still getting this disk activity peaks

Hi Dmitry,

I am also experiencing the same combination of errors. are you still getting this error resolved?

  1. Same issue. Things in common is that it’s a big database.

Does the hosted discourse have customers with big databases?

sudo du -h --max-depth=1 /var/discourse/shared/standalone/
28K     /var/discourse/shared/standalone/state
244K    /var/discourse/shared/standalone/postgres_run
8.0K    /var/discourse/shared/standalone/backups
1.2G    /var/discourse/shared/standalone/uploads
531M    /var/discourse/shared/standalone/log
4.0K    /var/discourse/shared/standalone/postgres_backup
12K     /var/discourse/shared/standalone/tmp
73M     /var/discourse/shared/standalone/redis_data
--> 89G     /var/discourse/shared/standalone/postgres_data <---
12K     /var/discourse/shared/standalone/ssl
91G     /var/discourse/shared/standalone/