User profile images loading slowly, failing to load

I’m also getting the 'hijack user_avatars show' is still running after 90 seconds on db default, this process may need to be restarted!

rebuilding from the command line made this problem go away for me… I left it for a few hours and it did not go away on its own.

This is not clearing up on it’s own for me. I did a rebuild and that did not help either. Here is the backtrace…

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.4/lib/logster/logger.rb:101:in `add_with_opts' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/logster-1.3.4/lib/logster/logger.rb:52:in `add' /usr/local/lib/ruby/2.5.0/logger.rb:545:in `error' /var/www/discourse/lib/scheduler/defer.rb:92:in `block (2 levels) in do_work' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0.pre.1/lib/message_bus/timer_thread.rb:110:in `do_work' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.2.0.pre.1/lib/message_bus/timer_thread.rb:38:in `block in initialize'

This issue should auto correct itself after a bit, we needed to rebuild all resized avatars so it could take a while to happen.

6 Likes

Thanks Sam. This does seem to be improving (but not yet finished) on our site. Just taking much longer than I’d think it would to rebuild avatars. Like, several hours for just a couple thousand users. That’s w/ the $10/mo Digital Ocean droplet…looks like it’s maxed out our CPU for going on 5 hours :sweat_smile:. Not sure if that’s about in line w/ what you’d expect, just FYI.

4 Likes

High CPU is very unlikely to be Avatar related, there was another change that was committed:

This can eat up the entire sidekiq queue for a while. It will clear up but depending on your post count it may take a bit.

9 Likes

Would it be a good idea to add this kind of events when you release the new beta version posts from #feature:announcements so that nobody will go OMG my forum iz now broken!!! PANIC!!! ?

Something like
Deploy attention:

  • because of <commit here>, after you upgrade your discourse all posts containing images will be rebaked so your forum will be a little slow for a while.
  • because of <meta post here>, after you upgrade avatars will be a little bit slow to show but it will go away in a few hours.
3 Likes

Yeah @jomaxro can add a note to next beta (which is when this officially lands) informing people to expect high CPU usage for a few hours after this upgrade due to image re-optimization.

I tweaked this today a fair bit to avoid queue starvation and issues where avatar resizing was taking way too long, but you should expect a reasonable CPU jump temporarily while stuff is catching up.

We basically needed to go to every post with images and rebake, then resize all the images. Plus resize all the avatars.

We did this cause our new algorithm for resizing is far more efficient. Plus you get retina image support and “fuzzy” image while it loads on old posts.

9 Likes

We have over 500,000 uploaded images that need to be reprocessed, so we’re looking at about 2 months of reprocessing. I’ll need to be very careful when deploying this change, I’m not too excited about it, tbh.

5 Likes

If you must you can run a 1 liner to disable the rebake

Great! Can you share that with me please?

I see the convert task is working away in htop still after 24 hours, but making progress. Is there any way to know how much longer this will take? Is there a way to de-prioritize the task so my site is not brought to its knees while this is going on? Quite a painful upgrade.

4 Likes

I usually update Discourse forum via command line.
Can you, please, specify, when avatars and images resizing tasks will be executed - during run of command sudo /var/discourse/launcher bootstrap app or after that - when I executed - sudo /var/discourse/launcher start app and app is running ?

Will this resize task be executed only during update to version 2.2.0.beta7, or it will be executed during updates to all next Discourse releases ?

After the /launcher start app it will run in the background while the app is running.

It’s a one time thing, however we trigger those rebakes whenever something big changes, like when we moved the markdown processor library.

Those are rare events, and we don’t have another one planned.

3 Likes

That sounds good! For any future things like this I wonder if it could be useful to prompt admins w/ a suggestion to beef up server resources temporarily before running.

Not sure what impact this would have, but for a big forum like @bartv’s could temporarily switching to e.g. a Digital Ocean droplet w/ tons more RAM / CPUs make this update process substantially faster?

1 Like

Hi,
After three days (v2.2.0.beta7 +13) there are a lot of missed user profiles (say 5-10% of 3000 users). How to check if it’s still processing something or should I apologize and ask users to upload profile image once again.

Thanks

Yeah I feel this is the best way for me, what’s the code?

Can you, please, tell me what triggers start of images rebake process ?
I started 1 instance with the latest version of Discourse and during 40 minutes convert command was not started.
ps aux command doesn’t show convert process.

As I understand convert process should do rebake of images.

Something along the lines of:

./launcher enter app
Post.where(baked_version: nil).update_all(baked_version: 2)

But this is not something I am going to support and I strongly recommend you test this on your testing instance first.

Instead I would just reduce rebake old posts count to a number you find acceptable, at 1 you will not notice any impact.

7 Likes

Another reason not to do this is that some future rebake you will want to take advantage of. At some point you’ll likely need to throw more resources to your server.

4 Likes