Potentially on-Topic: is MSNBot for real?

Sat Sep 23 00:09:59 UTC 2006

On Fri, 22 Sep 2006, Simon Waters wrote:

> On Friday 22 Sep 2006 11:39, you wrote:
>>
>> Is this unusual, or what?  Are search engines supposed to be amongst the
>> biggest user agents recorded on a typical website?  How much trolling and
>> indexing is considered 'too much' ?
>
> Whenever it becomes a problem.
>
> If you don't have enough genuine traffic, and you don't have much, then the
> search engines will look like they are dominating it, as they are pretty
> thorough.

I spose its all about scale.  In a country of 4 million odd people, a 
website with a domestic focus in a niche area - 40,000 hits in 21 days is 
'fair' IMHO.

> I've seen issues arise with some search bots, where they have discovered loops
> in a websites structure and downloaded multiple copies, or found novels links
> to dynamic content and indexed your entire database. So worth checking what
> pages they have been to, to see if those could be an issue.

Good point. Thanks for the pointer.

>
>> Off-list thoughts on this welcome if the operational relevance of this
>> issue is questioned...
>
> Trust me, anything involving 40,000 hits is off-topic in Nanog, unless you
> have reason to believe the same 40,000 are happening to everyone on the net,
> or they took down 40,000 important websites.

Seeing stats on sites much bigger than my own helps put perspective on it, 
so i'm already grateful for those who've responded.

>
> Most of the regular are just getting in, so expect to be flamed mercilessly.

Anythings gotta be better tham beating on Gadi, right?

=)

Mark.