'Updating Last Seen' and 'hijack topics timings' and 'Track Visit' is still running after 90 seconds

Dmitry_Krasnoperov · March 27, 2019, 10:09pm

I’ve got the problem.

Several time a day my forum goes in situation where all updates process are delayed.

'Updating Last Seen' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'Track Visit' is still running after 90 seconds on db default, this process may need to be restarted!
/var/www/discourse/plugins/discourse-topic-previews/lib/topic_list_previews_helper.rb:13:in `create_topic_thumbnails'
/var/www/discourse/plugins/discourse-topic-previews/lib/cooked_post_processor_edits.rb:76:in `update_post_image'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:40:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:10:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:35:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:24:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb:185:in `execute_job'

I see stange disk activity as well

or some peek on week

I have backup settings as “every 3 days”.
I do have two plugins
discourse-whos-online
discourse-topic-list-previews

What’s to check? Or logs to see?

Dmitry_Krasnoperov · March 27, 2019, 10:21pm

More errors:

Job exception: ERROR:  deadlock detected
DETAIL:  Process 22512 waits for ExclusiveLock on tuple (22418,9) of relation 16742 of database 16400; blocked by process 28074.
Process 28074 waits for ShareLock on transaction 1013165180; blocked by process 24391.
Process 24391 waits for ShareLock on transaction 1013163724; blocked by process 1150.
Process 1150 waits for ShareLock on transaction 1013156883; blocked by process 22512.
HINT:  See server log for query details.

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:118:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:90:in `exec'
(eval):6:in `exec'
/var/www/discourse/app/models/post.rb:646:in `block in calculate_avg_time'
/var/www/discourse/lib/freedom_patches/active_record_base.rb:45:in `retry_lock_error'
/var/www/discourse/app/models/post.rb:623:in `calculate_avg_time'
/var/www/discourse/app/jobs/scheduled/calculate_avg_time.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/app/jobs/base.rb:277:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'

codinghorror · March 28, 2019, 12:18am

Disable these and rebuild and see if that helps things.

pfaffman · March 28, 2019, 12:00pm

Do you have enough ram? I’d the cpu far enough?

How big is your database and how much ram do you have? Is swap enabled? Is it getting full?

Dmitry_Krasnoperov · March 29, 2019, 11:27pm

4CPU/8GB and 2GB swap enabled.
But DB is large enough with a lot of images

   84.0GiB [##########] /uploads
   26.7GiB [###       ] /postgres_data
    4.7GiB [          ] /backups
    3.7GiB [          ] /log
  201.5MiB [          ] /redis_data
    1.7MiB [          ] /letsencrypt
  176.0KiB [          ] /postgres_run
   28.0KiB [          ] /state
   16.0KiB [          ] /tmp
   12.0KiB [          ] /ssl
e   4.0KiB [          ] /postgres_backup

I’ve just got this frozen status update and htop shows

and i got Error
'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!

backtrace

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'

Dmitry_Krasnoperov · March 30, 2019, 12:06am

Yep, will disable one by one and see.
Thanks

pfaffman · March 30, 2019, 6:59am

That’s a big database. What’s the value of db_shared_buffers in app.yml?

Dmitry_Krasnoperov · March 30, 2019, 7:14am

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ##
  ## On 1GB installs set to 128MB (to leave room for other processes)
  ## on a 4GB instance you may raise to 1GB
  db_shared_buffers: "2048MB"

pfaffman · March 30, 2019, 12:02pm

You might try bumping it and the other memory variable up a bit. You have fast ssd?

Dmitry_Krasnoperov · May 19, 2019, 1:55pm

Hi,

I’m in DigitalOcean SSD disk.
Set db_shared_buffers: “3048MB”
Disable plugins one by one.

but I’m still getting this disk activity peaks

wandagoner · May 23, 2020, 2:54am

Hi Dmitry,

I am also experiencing the same combination of errors. are you still getting this error resolved?

Crius · October 9, 2024, 8:45am

Same issue. Things in common is that it’s a big database.

Does the hosted discourse have customers with big databases?

sudo du -h --max-depth=1 /var/discourse/shared/standalone/
28K     /var/discourse/shared/standalone/state
244K    /var/discourse/shared/standalone/postgres_run
8.0K    /var/discourse/shared/standalone/backups
1.2G    /var/discourse/shared/standalone/uploads
531M    /var/discourse/shared/standalone/log
4.0K    /var/discourse/shared/standalone/postgres_backup
12K     /var/discourse/shared/standalone/tmp
73M     /var/discourse/shared/standalone/redis_data
--> 89G     /var/discourse/shared/standalone/postgres_data <---
12K     /var/discourse/shared/standalone/ssl
91G     /var/discourse/shared/standalone/

Topic		Replies	Views
Stuck in a loop of freeing up space and filling it up again when rebuilding Installation server-resources	33	3635	June 8, 2024
Job exception: JavaScript was terminated (either by timeout or explicitly) Installation server-resources	8	986	October 14, 2019
Backups keep taking down my forum Bug backups	9	474	March 11, 2024
Hangs discourse and discourse restore FAILED Installation	24	1870	August 26, 2020
Precompiling assets takes 20 minutes Installation server-resources	18	900	January 31, 2024

'Updating Last Seen' and 'hijack topics timings' and 'Track Visit' is still running after 90 seconds

Related topics