That led me down the path of suspecting that it was crawler request at 9:40am. But I don’t know if there is a way to check that. I looked at the list of crawlers and the number of requests, most of them come from Google and Bing so for sure we aren’t going to block them.
That led me to the questions
is there a log where i can tell who’s accessing the site at a given point in time?
is there a way to have the crawlers spread out their requests since these are ‘good’ crawlers?
will throwing more CPU/ memory at the server help with the situation? I am a bit skeptical about that since CPU/ memory usage didn’t spike. is 80% average memory utilization too high?
We are at 2 vCPUs/ 2gb memory. We set the instance at 4 unicorn workers, which seems to be in line with the amount of ram that we have.
Yes, the slow down crawler user agents site setting.
Looks like what you had was I/O wait during the 9:40 peak. Increasing RAM may help as more stuff can be kept at cache, but I don’t know if that peak was read or write, as you did cut off the graph legend .
That said, if you can afford, increasing the droplet size to the next available size will always help.
Thanks. I checked out the log at 9:40am and the requests look like they come from user actions (consumer browsers) rather than crawlers.
The green peak was read.
Yeah, since it doesn’t look like its crawlers, I think I will trial with the next droplet size and increase RAM from 2gb to 4gb and see if it helps.
It is a little surprising to me that user activity is the cause, because I have always been under the impression that we have fewer active posters in the last 2 years than before. But when I looked at Google Analytics, we do have steadily increase users- perhaps even though posters have decreased, lurkers have increased…
.
