yahoo crawlers hammering us

Nathan Eisenberg nathan at atlasnetworks.us
Wed Sep 8 10:24:38 UTC 2010


> Possibly because that other user is who the customer pays have their content
> delivered to?

Customers don't want to deliver their content to search engines?  That seems silly.
 
http://www.last.fm/robots.txt (Note the final 3 disallow lines...)







More information about the NANOG mailing list