Uploading avatars slow or broken, with warnings in /logs

AI-generated summary

The issue of slow or broken avatar uploads with warnings in /logs has been reported by downey. The warnings include DistributedMutex errors and a hijack uploads error. The issue seemed to resolve itself, with downey reporting that avatars are now uploading and rendering quickly. codinghorror inquired about the version being upgraded from, and downey replied that it was from late last week. codinghorror suspects an arcane server problem, as there have been no other recent reports of this issue.

Today we noticed that it’s incredibly slow, if possible at all, to upload new avatar photos. It’s also been incredibly slow to load some (but not all) users’ larger avatars on the pop-up user cards or the full-page profile. Updated to 2.5.0.beta2 (ae3220fb88) hoping that would help but no luck.

In https://discourse.example.org/logs we see the following type of warning when trying to upload a new avatar image file through the user profile page:

DistributedMutex(“upload_2_TEST.jpg”): held for too long, expected max: 60 secs, took an extra 2 secs

/var/www/discourse/lib/distributed_mutex.rb:54:in `warn'
/var/www/discourse/lib/distributed_mutex.rb:37:in `ensure in block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:41:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'
/var/www/discourse/lib/upload_creator.rb:37:in `create_for'
/var/www/discourse/app/controllers/uploads_controller.rb:217:in `create_upload'
/var/www/discourse/app/controllers/uploads_controller.rb:36:in `block in create'
/var/www/discourse/lib/hijack.rb:56:in `instance_eval'
/var/www/discourse/lib/hijack.rb:56:in `block in hijack'
/var/www/discourse/lib/scheduler/defer.rb:94:in `block in do_work'
rails_multisite-2.1.0/lib/rails_multisite/connection_management.rb:64:in `with_connection'
/var/www/discourse/lib/scheduler/defer.rb:89:in `do_work'
/var/www/discourse/lib/scheduler/defer.rb:79:in `block (2 levels) in start_thread'

There are also some other items in /logs that seem to be related:

  • [Warning] DistributedMutex(“optimized_image_23_120_120”): held for too long, expected max: 60 secs, took an extra 5 secs
  • [Error] ‘hijack uploads create’ is still running after 90 seconds on db default, this process may need to be restarted!

Here’s what the UI looks like when attempting to upload an avatar image, after eventually timing out:

This seems to have self-resolved, I think … avatars are both uploading and rendering quickly now without errors.

Anyone ever seen something like this before? Is there some resource utilization I should be concerned about?

How old was the version you were upgrading from?

From late last week or so.

My guess is some kind of arcane server problem. No other reports of this recently.