yahoo crawlers hammering us

Wed Sep 8 16:20:07 UTC 2010

On Wed, Sep 08, 2010 at 12:04:07AM -0700, Matthew Petach said:

  >I *am* curious--what makes it any worse for a search engine like Google
  >to fetch the file than any other random user on the Internet?  In either case,
  >the machine doing the fetch isn't going to rate-limit the fetch, so
  >you're likely
  >to see the same impact on the machine, and on the bandwidth.

I think that the difference is that there's a way to get to Yahoo and ask them
WTF. Whereas the guy who mass downloads your site with a script in 2 hrs you
have no recourse to (modulo well funded banks dispatching squads with baseball
bats to resolve hacking incidents).  I also expect that Yahoo's behaviour is
driven by policy, not random assholishness (I hope :), and therefore I should
expect such incidents often. I also expect whinging on nanog might get me some
visiblity into said policy and leverage to change it! </dream>

/kc
-- 
Ken Chase - ken at heavycomputing.ca - +1 416 897 6284 - Toronto CANADA
Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.