Getting deadlock errors in logs

Product Support

AI-generated summary

Users are experiencing deadlock errors in the logs related to the ActivityPub plugin, specifically during delivery jobs. pmusaraj reports a PG::TRDeadlockDetected error involving the discourse_activity_pub_actors table, with the stack trace pointing to a job handling deliveries. The issue appears to stem from concurrent updates to the available column of actors, particularly when dealing with large datasets—such as 90k+ actors for mastodon.social. angus initially suggests a fix, but pmusaraj notes it’s insufficient due to the scale of data and proposes that updating the available column on every delivery may not be necessary. angus responds with another proposed fix to address the issue.

1 
Job exception: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 29659 waits for ShareLock on transaction 892416679; blocked by process 29650.
Process 29650 waits for ShareLock on transaction 892416541; blocked by process 29659.
HINT:  See server log for query details.
CONTEXT:  while rechecking updated tuple (36923,3) in relation "discourse_activity_pub_actors"

CleanShot 2025-03-27 at 08.57.49@2x
CleanShot 2025-03-27 at 08.57.49@2x1920×500 96.5 KB

These are showing up in /logs @angus, I’m not sure if there are any user-facing issues. Stack trace points to a job handling deliveries:

rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:69:in `exec_params'  
rack-mini-profiler-3.3.1/lib/patches/db/pg.rb:69:in `exec_params'  
(eval at /var/www/discourse/lib/method_profiler.rb:38):105:in `exec_params'
activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:894:in `block (2 levels) in exec_no_cache'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:1004:in `block in with_raw_connection'  
activesupport-7.2.2.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:976:in `with_raw_connection'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:893:in `block in exec_no_cache'  
activesupport-7.2.2.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:1119:in `log'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:892:in `exec_no_cache'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_delete'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:208:in `update'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in `update'  
activerecord-7.2.2.1/lib/active_record/relation.rb:610:in `block in update_all'  
activerecord-7.2.2.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:421:in `with_connection'  
activerecord-7.2.2.1/lib/active_record/connection_handling.rb:296:in `with_connection'  
activerecord-7.2.2.1/lib/active_record/relation.rb:598:in `update_all'  
/var/www/discourse/plugins/discourse-activity-pub/lib/discourse_activity_pub/delivery_failure_tracker.rb:36:in `set_domain_actors_as_available'  
/var/www/discourse/plugins/discourse-activity-pub/lib/discourse_activity_pub/delivery_failure_tracker.rb:22:in `track_success'  
/var/www/discourse/plugins/discourse-activity-pub/app/jobs/discourse_activity_pub_deliver.rb:50:in `perform_request'  
/var/www/discourse/plugins/discourse-activity-pub/app/jobs/discourse_activity_pub_deliver.rb:15:in `execute'  
/var/www/discourse/app/jobs/base.rb:316:in `block (2 levels) in perform'
3

This should address it

https://github.com/discourse/discourse-activity-pub/pull/203

4

Hmm, that doesn’t seem to fix the issue unfortunately @angus. There’s a lot of actors, possibly, in the DB, and the batching is still insufficient.

On meta, for example, there are 90k+ rows in that model:

discourse(prod)> DiscourseActivityPubActor.where(domain: 'mastodon.social').size
=> 90264

Do we absolutely need to update their available column on every delivery?

5

I see. Hopefully this, together with the previous changes, will address it:

https://github.com/discourse/discourse-activity-pub/pull/204