Potentially on-Topic: is MSNBot for real?
blakjak at blakjak.net
Sat Sep 23 00:09:59 UTC 2006
On Fri, 22 Sep 2006, Simon Waters wrote:
> On Friday 22 Sep 2006 11:39, you wrote:
>> Is this unusual, or what? Are search engines supposed to be amongst the
>> biggest user agents recorded on a typical website? How much trolling and
>> indexing is considered 'too much' ?
> Whenever it becomes a problem.
> If you don't have enough genuine traffic, and you don't have much, then the
> search engines will look like they are dominating it, as they are pretty
I spose its all about scale. In a country of 4 million odd people, a
website with a domestic focus in a niche area - 40,000 hits in 21 days is
> I've seen issues arise with some search bots, where they have discovered loops
> in a websites structure and downloaded multiple copies, or found novels links
> to dynamic content and indexed your entire database. So worth checking what
> pages they have been to, to see if those could be an issue.
Good point. Thanks for the pointer.
>> Off-list thoughts on this welcome if the operational relevance of this
>> issue is questioned...
> Trust me, anything involving 40,000 hits is off-topic in Nanog, unless you
> have reason to believe the same 40,000 are happening to everyone on the net,
> or they took down 40,000 important websites.
Seeing stats on sites much bigger than my own helps put perspective on it,
so i'm already grateful for those who've responded.
> Most of the regular are just getting in, so expect to be flamed mercilessly.
Anythings gotta be better tham beating on Gadi, right?
More information about the NANOG