Applebot has been crawling my Discourse site like crazy for the past couple of months, and I’m not totally sure why.
Back in June, I wasn’t originally concerned, since it seemed more likely that Apple would buy a search engine like DuckDuckGo rather than ever make their own search engine.
Then I wondered what Apple could possibly be doing. Other developers have reported the same thing regarding Applebot back in August.
More recently, there’s been an incident with a report by the Financial Times in late October, leaving me fairly concerned that I was unprepared for something major.
With all of that said, let me just put into perspective that I’m getting around 100,000 to 200,000 pageviews a week from Applebot alone, which is hammering my Discourse site in terms of performance, as Applebot isn’t being throttled whatsoever.
Please tell me what I should do. The amount of pageviews from Applebot is worrying.