Uploads timing out or very slow

Support
1

In the past couple of weeks, we have started noticing extremely slow upload times on our site for attachements, and sometimes the upload even times out (see related logs below). My setup is digital ocean space for s3, and bunny.net for CDN. I also turned on the new composer media optimization image enabled, but I think it was happening before this. It’s a bit hit and miss trying to repro this problem. Any ideas?

Timeout log 
DistributedMutex("optimized_image_32792_50_33"): held for too long, expected max: 60 secs, took an extra 66 secs

Message (122 copies reported)

DistributedMutex("optimized_image_32792_50_33"): held for too long, expected max: 60 secs, took an extra 66 secs

Backtrace

/var/www/discourse/lib/distributed_mutex.rb:54:in `warn'
/var/www/discourse/lib/distributed_mutex.rb:37:in `block in synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:29:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:14:in `synchronize'
/var/www/discourse/app/models/optimized_image.rb:18:in `lock'
/var/www/discourse/app/models/optimized_image.rb:68:in `create_for'
/var/www/discourse/app/models/topic_thumbnail.rb:21:in `find_or_create_for!'
/var/www/discourse/app/models/topic.rb:104:in `block in generate_thumbnails!'
/var/www/discourse/app/models/topic.rb:103:in `each'
1 Like
2

Is your server CPU starved?

By the way this has nothing to do with either S3/CDN/Composer Optimization as this is the usual server side image processing pipeline.

3

Nope, for the past week haven’t exceed 60% cpu, 50% memory, 30% disk…

4

Still having major issues with this, tried updating to latest and using safe mode which did not help. I tried turning off enable s3 uploads and the problem does go away. Any ideas?

5

I recommend you first try out Amazon S3

If that resolves the issue, the bug is a digital ocean bug

3 Likes
6

Digital ocean had a huge issue with S3 in Amsterdam a while back (which I started using because a friend who worked there said it was their best data center, but things change and now he works for Google).

So another S3 service was going to be my suggestion too.

https://status.digitalocean.com/ shows issues with spaces in sfo2 for the past several days. It says “resolved” but I’m not recommending spaces anymore…

3 Likes
7

Yes, that is right. This is why I moved to Frankfurt. Since then works fine without any problem. AMS3 and SFO2 has a lots of issue, maintance and availability problem always.

2 Likes
8

Thanks guys, the DO space in San Fransico does seem to be the issue, I switched over to New York for now and may consider going back to Amazon if I have any more trouble. How do I go about remapping existing images over to the new space? Do I just do a rebake? I have copied all the files over to the new space.