Potentially on-Topic: is MSNBot for real?

Fri Sep 22 10:39:34 UTC 2006

On a website I host with nearly 9000 unique visits month-to-date (thats 
visits, not hits) a full 20% of the recorded 'hits' (Hitcount is ~40,000) 
are being generated by 'msnbot'.  We see this as a large amount of http 
traffic from IP addresses owned by Microsoft.

I've actually seen this across a number of websites (including my own) but 
the guest on my server has raised the issue of loading being completely 
misproportionate to the perceived value of the visit - and asked about 
potentially blocking them off entirely.

Is this unusual, or what?  Are search engines supposed to be amongst the 
biggest user agents recorded on a typical website?  How much trolling and 
indexing is considered 'too much' ?

At what point to the search engines themselves become a menace - the load 
the cause outweighs the value of said load?  (I'd like my cpu cycles to be 
for real people, please...)

Off-list thoughts on this welcome if the operational relevance of this 
issue is questioned...

Cheers
Mark.