I’ve just added my forum to Google Webmaster Tools, and saw a bunch of
502 Bad Gateway errors as a result of the first crawl. My forum has very little utilization now (a couple hundred visitors a day), plus it was Friday 7pm Pacific (where most users are).
My host is a Linode with 1GB of RAM and 2.5GB of swap and I had upgraded to the latest Discourse.
Has anyone else seen 502 Google bot crawl errors with a similar configuration? You can check by going to Crawl errors and see if there are any errors in the Server tab.
Is this performance expected of Discourse and should I limit the crawl rate, or might I have something wrong in my config (I followed the 30-minute install)?