Google allows webmasters to control the 'speed' (or more accurately 'ferocity') with which it crawls your site. For some reason that setting (which I set to 'slow') expires on it's own and resets itself to 'normal' which is more than our servers can handle.
Our configuration database was corrupted a few minutes after the first 'normal' crawl started. 16 hours laters Google bot was STILL banging away acting surprisingly like a very ill behaved spam bot (hitting the same pages multiple times, ignoring 100,000s 'access denied' messages we threw at it and ignoring our 'do not search' insructions they claim to honor).
I was finally able to isolate them and work on the actual corruption problem.
The drag is we run the risk of being "de listed" on google which would be really bad site - almost as bad as being taken down by their abuse.
Posts
4 comments:
That was a long one. What happened?
Google allows webmasters to control the 'speed' (or more accurately 'ferocity') with which it crawls your site. For some reason that setting (which I set to 'slow') expires on it's own and resets itself to 'normal' which is more than our servers can handle.
Our configuration database was corrupted a few minutes after the first 'normal' crawl started. 16 hours laters Google bot was STILL banging away acting surprisingly like a very ill behaved spam bot (hitting the same pages multiple times, ignoring 100,000s 'access denied' messages we threw at it and ignoring our 'do not search' insructions they claim to honor).
I was finally able to isolate them and work on the actual corruption problem.
The drag is we run the risk of being "de listed" on google which would be really bad site - almost as bad as being taken down by their abuse.
damn. that sucks.
I didn't realize the server was so maxed out. Also didn't realize Google was so aggressive. Sounds like a trojan virus.
I'm glad it got sorted.
Post a Comment