Google crawler errors / Discourse forum down

(Dan Dascalescu) #1

I’ve just added my forum to Google Webmaster Tools, and saw a bunch of 502 Bad Gateway errors as a result of the first crawl. My forum has very little utilization now (a couple hundred visitors a day), plus it was Friday 7pm Pacific (where most users are).

My host is a Linode with 1GB of RAM and 2.5GB of swap and I had upgraded to the latest Discourse.

Has anyone else seen 502 Google bot crawl errors with a similar configuration? You can check by going to Crawl errors and see if there are any errors in the Server tab.

Is this performance expected of Discourse and should I limit the crawl rate, or might I have something wrong in my config (I followed the 30-minute install)?

(Jeff Atwood) #2

I am not aware of this happening anywhere else. Is your setup Docker based? Did you override Googles crawl rate settings? (Not sure if Google even allows this, anyhow)

(Dan Dascalescu) #3

I followed the Docker setup, yes.

Haven’t touched any crawl settings; was wondering if I should. You can control the crawl rate via Search Console -> Site Settings:

(Jeff Atwood) #4

I am not aware of anyone needing to change the Google defaults here.

(Dan Dascalescu) #5

That means my Discourse instance couldn’t handle the load. I’m curious if anyone else has seen this problem. One can go to Crawl errors and see if there are any server errors.

(Sam Saffron) #6

I wonder if this is the rate limiting template…

(Jeff Atwood) #7

Could be Google is just auto-diagnosing the correct crawl rate. Not sure there’s an actual problem here.