Massive Amount of Traffic from Applebot Crawler

Applebot has been crawling my Discourse site like crazy for the past couple of months, and I’m not totally sure why.

Back in June, I wasn’t originally concerned, since it seemed more likely that Apple would buy a search engine like DuckDuckGo rather than ever make their own search engine.


Then I wondered what Apple could possibly be doing. Other developers have reported the same thing regarding Applebot back in August.

More recently, there’s been an incident with a report by the Financial Times in late October, leaving me fairly concerned that I was unprepared for something major.

With all of that said, let me just put into perspective that I’m getting around 100,000 to 200,000 pageviews a week from Applebot alone, which is hammering my Discourse site in terms of performance, as Applebot isn’t being throttled whatsoever.

Please tell me what I should do. The amount of pageviews from Applebot is worrying.

2 Likes

You can block Applebot user agent on the site settings.

4 Likes

I don’t want to block the Applebot crawler entirely though.

2 Likes

Add it to the slow down crawler user agents site setting?

10 Likes

I would do that, but I just want to know why Applebot is so aggressively crawling my site. With wishful thinking, could it just be for Spotlight search results and Siri search results, or is it something more extreme? Now I hope that my site is being indexed correctly by Applebot.

Is anyone else having the same experience as me?

2 Likes

Yeah, howdy. Signed up to say that I’m experiencing something similar and I’m also concerned. Getting around 150000 views from applebot last I checked. I’m not sure about the whole Apple search engine thing due to Google paying them billions annually though, but honestly, who knows? Maybe that analyst in the first article you linked is right (it then led me to a different article that I linked below).


Overall, I’m hesitant to slow down applebot or throttle it or whatever due to the circumstances mentioned in this topic. My community has had enough trouble lately.
4 Likes

Thank you for the reply! Good thing I’m not alone. Wondering how to handle the situation correctly is not easy.

slow down crawler user agents is a last ditch effort, but I’m not going to give up trying to find an alternative solution or a solid reason as to why Applebot has become so aggressive all of the sudden.

2 Likes