Spammer web harvesting tool countermeasures

Jon Stevens jon at
Fri Oct 31 04:01:44 UTC 1997

"Deepak Jain" <deepak at> said the following at 10/30/97 6:56 PM:

>And wouldn't we, in turn, see some kind of problems arise with legitimate 
>search engines because of this?

If you downloaded it and looked at it, you would have noticed that it 
follows search engine guidelines by adding the appropriate <META> tag to 
the HTML as well as the fact, that you can also use the robots.txt file 
to block it.

Of course this also breaks down if spammer robots actually follow the 
rules...but how many of those do you think that there are? ;-)


Jon (no h) S. Stevens
Web Engineer
j at
Clear Ink and The Internet Weather Report
<> | <>

More information about the NANOG mailing list