1.8M PullHotLinkedImages in our sidekiq backlog


(Paulo Silveira) #1

We are trying to cope with a huge batch of PullHotLinkedImages (and ProcessoPost/UpdateGravatar/CrawlLink) in our sidekiq queue after migrating our forum. And as the posts are being processed, the queue is getting even bigger!

To be honest, I dont even understand why we have so many PullHotLinkedImages/etc, since our posts have almost no images attached. Could/should we just remove all these jobs?

Any hints?


(Felix Freiberger) #2

This looks normal – Discourse needs some time to process these posts. You might consider increasing the number of workers in the container definition if you can dedicate more system resources for it.


(Paulo Silveira) #3

would it be a problem if we just delete the queue? what would we lose not processing these old posts?


(Régis Hanol) #4

Not much. PullHotlinkedImages will only try to download external images in order to prevent link rot. But that might be fine in your case.


(Paulo Silveira) #5

@zogstrip one last question: the other jobs are also optional right? like ProcessPost/CrawlExternalLink? and is there any way to disable these services?


(Régis Hanol) #6

These other jobs aren’t required but they definitely enhance the presentations of your old posts. But if you want, you can “rebake” your posts later either manually or automatically via a rake task.