The discussion revolves around a large backlog of Sidekiq jobs, including PullHotLinkedImages, ProcessPost, and CrawlExternalLink, following a forum migration. Paulo_Silveira is concerned about the volume of these jobs, especially since their posts have few images. fefrei suggests it’s normal and recommends increasing workers if more resources are available. zogstrip reassures that deleting the queue won’t cause significant issues, as PullHotlinkedImages mainly prevents link rot and the other jobs enhance post presentation but aren’t essential. Paulo_Silveira asks if these jobs can be disabled, and zogstrip confirms they can be skipped, with the option to rebake posts later via a rake task.
We are trying to cope with a huge batch of PullHotLinkedImages (and ProcessoPost/UpdateGravatar/CrawlLink) in our sidekiq queue after migrating our forum. And as the posts are being processed, the queue is getting even bigger!
To be honest, I dont even understand why we have so many PullHotLinkedImages/etc, since our posts have almost no images attached. Could/should we just remove all these jobs?
Any hints?
This looks normal – Discourse needs some time to process these posts. You might consider increasing the number of workers in the container definition if you can dedicate more system resources for it.
would it be a problem if we just delete the queue? what would we lose not processing these old posts?
Not much. PullHotlinkedImages will only try to download external images in order to prevent link rot. But that might be fine in your case.
@zogstrip one last question: the other jobs are also optional right? like ProcessPost/CrawlExternalLink? and is there any way to disable these services?
These other jobs aren’t required but they definitely enhance the presentations of your old posts. But if you want, you can “rebake” your posts later either manually or automatically via a rake task.
