yahoo crawlers hammering us

Wed Sep 8 14:07:46 UTC 2010

On Wed, 08 Sep 2010 02:21:31 PDT, Bruce Williams said:
> > I *am* curious--what makes it any worse for a search engine like Google
> > to fetch the file than any other random user on the Internet
> 
> Possibly because that other user is who the customer pays have their
> content delivered to?

Seems to me that if you're doing content-for-pay and are upset when
some(body|thing) downloads it without paying for it, you shouldn't be leaving
gigabytes of said content where even a stupid bot can find it and download it
without paying for it.  Just sayin'.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20100908/c276dece/attachment.sig>