“最后出现时间”、“劫持主题时间”和“跟踪访问”仍在90秒后运行

[Dmitry_Krasnoperov]

我遇到了问题。

每天多次，我的论坛会进入所有更新进程都被延迟的状态。

'更新最后访问时间' 在数据库默认设置下已运行超过 90 秒，此进程可能需要重启！
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'劫持主题时间统计' 在数据库默认设置下已运行超过 90 秒，此进程可能需要重启！
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.1.2/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'


'跟踪访问' 在数据库默认设置下已运行超过 90 秒，此进程可能需要重启！
/var/www/discourse/plugins/discourse-topic-previews/lib/topic_list_previews_helper.rb:13:in `create_topic_thumbnails'
/var/www/discourse/plugins/discourse-topic-previews/lib/cooked_post_processor_edits.rb:76:in `update_post_image'
/var/www/discourse/lib/cooked_post_processor.rb:42:in `block in post_process'
/var/www/discourse/lib/distributed_mutex.rb:40:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:10:in `synchronize'
/var/www/discourse/lib/cooked_post_processor.rb:35:in `post_process'
/var/www/discourse/app/jobs/regular/process_post.rb:24:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.2.5/lib/sidekiq/processor.rb:185:in `execute_job'

我还注意到异常的磁盘活动

或者某周的峰值

我的备份设置是“每 3 天一次”。
我安装了两个插件：
discourse-whos-online
discourse-topic-list-previews

我应该检查什么？或者查看哪些日志？

[Dmitry_Krasnoperov]

More errors:

Job exception: ERROR:  deadlock detected
DETAIL:  Process 22512 waits for ExclusiveLock on tuple (22418,9) of relation 16742 of database 16400; blocked by process 28074.
Process 28074 waits for ShareLock on transaction 1013165180; blocked by process 24391.
Process 24391 waits for ShareLock on transaction 1013163724; blocked by process 1150.
Process 1150 waits for ShareLock on transaction 1013156883; blocked by process 22512.
HINT:  See server log for query details.

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.2/lib/patches/db/pg.rb:110:in `async_exec'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:118:in `run'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_sql-0.2.1/lib/mini_sql/postgres/connection.rb:90:in `exec'
(eval):6:in `exec'
/var/www/discourse/app/models/post.rb:646:in `block in calculate_avg_time'
/var/www/discourse/lib/freedom_patches/active_record_base.rb:45:in `retry_lock_error'
/var/www/discourse/app/models/post.rb:623:in `calculate_avg_time'
/var/www/discourse/app/jobs/scheduled/calculate_avg_time.rb:10:in `execute'
/var/www/discourse/app/jobs/base.rb:230:in `block (2 levels) in perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.6/lib/rails_multisite/connection_management.rb:63:in `with_connection'
/var/www/discourse/app/jobs/base.rb:219:in `block in perform'
/var/www/discourse/app/jobs/base.rb:215:in `each'
/var/www/discourse/app/jobs/base.rb:215:in `perform'
/var/www/discourse/app/jobs/base.rb:277:in `perform'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:82:in `process_queue'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mini_scheduler-0.9.1/lib/mini_scheduler/manager.rb:30:in `block in initialize'

[codinghorror (Jeff Atwood)]

@Dmitry_Krasnoperov

I do have two plugins
discourse-whos-online
discourse-topic-list-previews

Disable these and rebuild and see if that helps things.

[pfaffman (Jay Pfaffman)]

Do you have enough ram? I’d the cpu far enough?

How big is your database and how much ram do you have? Is swap enabled? Is it getting full?

[Dmitry_Krasnoperov]

4CPU/8GB and 2GB swap enabled.
But DB is large enough with a lot of images

   84.0GiB [##########] /uploads
   26.7GiB [###       ] /postgres_data
    4.7GiB [          ] /backups
    3.7GiB [          ] /log
  201.5MiB [          ] /redis_data
    1.7MiB [          ] /letsencrypt
  176.0KiB [          ] /postgres_run
   28.0KiB [          ] /state
   16.0KiB [          ] /tmp
   12.0KiB [          ] /ssl
e   4.0KiB [          ] /postgres_backup

I’ve just got this frozen status update and htop shows

and i got Error
'hijack topics timings' is still running after 90 seconds on db default, this process may need to be restarted!

backtrace

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:110:in `report_to_store'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:101:in `add_with_opts'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-2.3.0/lib/logster/logger.rb:52:in `add'
/usr/local/lib/ruby/2.5.0/logger.rb:545:in `error'
/var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:110:in `do_work'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0/lib/message_bus/timer_thread.rb:38:in `block in initialize'

[Dmitry_Krasnoperov]

@codinghorror

Disable these and rebuild and see if that helps things.

Yep, will disable one by one and see.
Thanks

[pfaffman (Jay Pfaffman)]

That’s a big database. What’s the value of db_shared_buffers in app.yml?

[Dmitry_Krasnoperov]

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ##
  ## On 1GB installs set to 128MB (to leave room for other processes)
  ## on a 4GB instance you may raise to 1GB
  db_shared_buffers: "2048MB"

[pfaffman (Jay Pfaffman)]

You might try bumping it and the other memory variable up a bit. You have fast ssd?

[Dmitry_Krasnoperov]

Hi,

I’m in DigitalOcean SSD disk.
Set db_shared_buffers: “3048MB”
Disable plugins one by one.

but I’m still getting this disk activity peaks

[wandagoner]

你好，德米特里，

我也遇到了同样的错误组合。你解决这个错误了吗？

[Crius]

2024年。同样的问题。共同点是它是一个大型数据库。

托管的 Discourse 是否有客户拥有大型数据库？

sudo du -h --max-depth=1 /var/discourse/shared/standalone/
28K     /var/discourse/shared/standalone/state
244K    /var/discourse/shared/standalone/postgres_run
8.0K    /var/discourse/shared/standalone/backups
1.2G    /var/discourse/shared/standalone/uploads
531M    /var/discourse/shared/standalone/log
4.0K    /var/discourse/shared/standalone/postgres_backup
12K     /var/discourse/shared/standalone/tmp
73M     /var/discourse/shared/standalone/redis_data
--
89G     /var/discourse/shared/standalone/postgres_data <---
12K     /var/discourse/shared/standalone/ssl
91G     /var/discourse/shared/standalone/