What does Jobs::Crawltopiclink in Sidekiq (Discourse) mean?

support
1

What does jobs::crawltopiclink in sidekiq mean?
In discourse, what is jobs::crawltopiclink used for?
Jobs::Crawltopiclink affect website SEO?

2

You can look at the source to see.

Is there some problem you are trying to solve?

3

Actually, I’m looking for the meaning of Jobs::Crawltopiclink on discourse (shown in Sidekiq queue).
I was looking for the meaning of Jobs::Crawltopiclink , then decided to delete the queue Jobs::Crawltopiclink .
But I didn’t find much documentation about Jobs::Crawltopiclink .
So I asked the discourse community
Thanks

4

But there are dozens of such jobs. Why did you think you cared about this one?

I think that if a topic title is a link, it crawls it.

1 Like
5

thanks you,
Before submitting the question, I also looked at the documentation via the link: discourse/crawl_topic_link.rb at main · discourse/discourse · GitHub
But really, I don’t understand the meaning of Jobs::Crawltopiclink in sidekiq (discourse)

our low queue (sidekiq) has multiple tasks: Jobs::Crawltopiclink ,
if I don’t clear the Jobs::Crawltopiclink queue, I’ll need more time to run it all, (while time is important, I don’t want to waste any more time)
I wonder if I should delete the Jobs::Crawltopiclink queue?

So I want to know the meaning of Jobs::Crawltopiclink.

Your answer may not be correct, hope to get more answers from you and everyone

1 Like
6

Unless I’m missing something, TopicLink appears to handle any link that appears in a post. When one is created, it enqueues that crawl_topic_link job which then crawls the URL to retrieve the title.

For example, I could enter exactly the following into a post:

This will be a link: https://github.com/discourse/discourse/blob/main/app/jobs/regular/crawl_topic_link.rb

Initially, that is how it will be baked into the post, displaying the URL as a link. When that job runs, it will crawl the URL to retrieve the title, then the post will be rebaked to display the title as a link in the way it appears in your previous post.

Normally this will happen very quickly so it will be barely noticeable but if you have a lot of those jobs queued up for some reason, you might see URLs for prolonged periods before they are replaced with the titles. I expect that if you delete the jobs, they will remain as URLs instead of titled links.

If you have recently performed an import, so created a lot of new posts in a very short space of time, that may explain a backlog of such jobs. I think you will have the best result by allowing those jobs to finish. I believe this to be a good default position with Sidekiq: just let it get on with what it does. (Unless you are seeing specific problems with some jobs.)

3 Likes
7

@Simon_Manning
thank you very much,
I partially understood the meaning of jobs::crawltopiclink

You have a very broad understanding.

2 Likes