Spammer web harvesting tool countermeasures
Deepak Jain
deepak at jain.com
Fri Oct 31 04:27:03 UTC 1997
I didn't download it, but I looked at the first page. I figured that if
it relied on someone setting up robots.txt correctly, there would be a
lot of people who don't do it correctly and we'll see installations of
the thing slow down search engines w/o good controls. Auto Meta Tags would
certainly help, except the next generation web scrapers will be set to
ignore them too.
-Deepak.
On Thu, 30 Oct 1997, Jon Stevens wrote:
> "Deepak Jain" <deepak at jain.com> said the following at 10/30/97 6:56 PM:
>
> >And wouldn't we, in turn, see some kind of problems arise with legitimate
> >search engines because of this?
>
> If you downloaded it and looked at it, you would have noticed that it
> follows search engine guidelines by adding the appropriate <META> tag to
> the HTML as well as the fact, that you can also use the robots.txt file
> to block it.
>
> Of course this also breaks down if spammer robots actually follow the
> rules...but how many of those do you think that there are? ;-)
>
> -jon
>
> Jon (no h) S. Stevens
> Web Engineer
> j at clearink.com
> Clear Ink and The Internet Weather Report
> <http://www.clearink.com/> | <http://www.internetweather.com/>
>
>
More information about the NANOG
mailing list