[2.4.0beta9] ActiveRecord::Deadlocked (PG::TRDeadlockDetected: ERROR: deadlock detected)

Hi all, after upgrading to latest beta, we started getting this error. As it seems from the environment it is related to converting a topic to a private message. Actually when we did this conversion, everything crashed and we received several 500 errors, and we had to refresh the page for it to work after a while.

Any ideas? :slight_smile:

Message

ActiveRecord::Deadlocked (PG::TRDeadlockDetected: ERROR: deadlock detected DETAIL: Process 11719 waits for ShareLock on transaction 25493716; blocked by process 11797. Process 11797 waits for ExclusiveLock on tuple (574,237) of relation 17882 of database 16384; blocked by process 11890. Process 11890 waits for ShareLock on transaction 25493702; blocked by process 11719. HINT: See server log for query details. CONTEXT: while updating tuple (493,14) in relation "topics" ) /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.4/lib/patches/db/pg.rb:72:in `async_exec_params'

Backtrace

/var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.4/lib/patches/db/pg.rb:72:in `async_exec_params' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.4/lib/patches/db/pg.rb:72:in `exec_params' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:718:in `block (2 levels) in log' /usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:717:in `block in log' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract_adapter.rb:708:in `log' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `execute_and_clear' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:111:in `exec_delete' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:174:in `update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/persistence.rb:385:in `_update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/persistence.rb:897:in `_update_row' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/locking/optimistic.rb:79:in `_update_row' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/persistence.rb:919:in `_update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/attribute_methods/dirty.rb:205:in `_update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/callbacks.rb:336:in `block in _update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:135:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:827:in `_run_update_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/callbacks.rb:336:in `_update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/timestamp.rb:124:in `_update_record' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/persistence.rb:906:in `create_or_update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/callbacks.rb:328:in `block in create_or_update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:135:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:827:in `_run_save_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/callbacks.rb:328:in `create_or_update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/timestamp.rb:129:in `create_or_update' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/persistence.rb:470:in `save' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/validations.rb:47:in `save' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:315:in `block in save' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:375:in `block in with_transaction_returning_status' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:279:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:366:in `with_transaction_returning_status' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:315:in `save' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/suppressor.rb:44:in `save' /var/www/discourse/lib/post_revisor.rb:442:in `block in update_topic' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:279:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction' /var/www/discourse/lib/post_revisor.rb:434:in `update_topic' /var/www/discourse/lib/post_revisor.rb:336:in `revise' /var/www/discourse/lib/post_revisor.rb:242:in `revise_post' /var/www/discourse/lib/post_revisor.rb:161:in `block in revise!' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:279:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction' /var/www/discourse/lib/post_revisor.rb:160:in `revise!' /var/www/discourse/app/models/topic_converter.rb:45:in `block in convert_to_private_message' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `block in transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction' /usr/local/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:281:in `transaction' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/transactions.rb:212:in `transaction' /var/www/discourse/app/models/topic_converter.rb:42:in `convert_to_private_message' /var/www/discourse/app/models/topic.rb:1333:in `convert_to_private_message' /var/www/discourse/app/controllers/topics_controller.rb:879:in `convert_topic' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/abstract_controller/base.rb:196:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/rendering.rb:30:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:135:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/abstract_controller/callbacks.rb:41:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/rescue.rb:22:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications.rb:180:in `block in instrument' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/notifications.rb:180:in `instrument' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-6.0.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/abstract_controller/base.rb:136:in `process' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-6.0.1/lib/action_view/rendering.rb:39:in `process' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.4/lib/mini_profiler/profiling_methods.rb:104:in `block in profile_method' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal.rb:191:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_controller/metal.rb:252:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/routing/route_set.rb:51:in `dispatch' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/routing/route_set.rb:33:in `serve' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/journey/router.rb:49:in `block in serve' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/journey/router.rb:32:in `each' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/journey/router.rb:32:in `serve' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/routing/route_set.rb:837:in `call' /var/www/discourse/lib/middleware/omniauth_bypass_middleware.rb:68:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/conditional_get.rb:38:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call' /var/www/discourse/lib/content_security_policy/middleware.rb:12:in `call' /var/www/discourse/lib/middleware/anonymous_cache.rb:318:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:259:in `context' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/session/abstract/id.rb:253:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/cookies.rb:648:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:101:in `run_callbacks' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.5.1/lib/logster/middleware/reporter.rb:43:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.1/lib/rails/rack/logger.rb:38:in `call_app' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.1/lib/rails/rack/logger.rb:28:in `call' /var/www/discourse/config/initializers/100-quiet_logger.rb:18:in `call' /var/www/discourse/config/initializers/100-silence_logger.rb:31:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/request_id.rb:27:in `call' /var/www/discourse/lib/middleware/enforce_hostname.rb:17:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/method_override.rb:22:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/executor.rb:14:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/sendfile.rb:111:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-6.0.1/lib/action_dispatch/middleware/host_authorization.rb:77:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.4/lib/mini_profiler/profiler.rb:296:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.3/lib/message_bus/rack/middleware.rb:57:in `call' /var/www/discourse/lib/middleware/request_tracker.rb:181:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.1/lib/rails/engine.rb:526:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `public_send' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `method_missing' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:68:in `block in call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `each' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `call' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:605:in `process_client' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:700:in `worker_loop' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:548:in `spawn_missing_workers' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/lib/unicorn/http_server.rb:144:in `start' /var/www/discourse/vendor/bundle/ruby/2.6.0/gems/unicorn-5.5.1/bin/unicorn:128:in `<top (required)>' /var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load' /var/www/discourse/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<main>'

Env

hostname physiart-app
process_id 19471
application_version 7737ecd0fed808a1c0e429565058c817af99d707
HTTP_HOST rembetiko.gr
REQUEST_URI /t/38631/convert-topic/private
REQUEST_METHOD PUT
HTTP_USER_AGENT Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
HTTP_ACCEPT /
HTTP_REFERER https://rembetiko.gr/t/τακτική-συνάντηση-παρασκευή-3-ιανουαρίου-2020/38631/23
HTTP_X_FORWARDED_FOR 2a02:587:3012:8800:2474:55f4:b2b3:f0a3, 162.158.159.43, 2a02:587:3012:8800:2474:55f4:b2b3:f0a3
HTTP_X_REAL_IP 2a02:587:3012:8800:2474:55f4:b2b3:f0a3
username chrispanag
1 Like

Can you check on Monday if you can repro any problems with converting a topic to a PM @osioke?

5 Likes

Sure thing, I’ll check on Monday.

4 Likes

Checking through this and trying to repro, I didn’t notice or see any problems. No obvious errors in my console and converting worked seamlessly.

Talking to @dan, this error most likely is as a result of concurrent processes and so the deadlock happens. Basically, there are two processes waiting for each other and no one can progress because of the other.

This could be something like “a user posted and I tried to convert a topic at exactly the same time”.

@dan could not reproduce the issue either.

@chrispanag do you have any custom plugins installed? Those could also interfere.

1 Like