Monitoring other people's sites (Was: Website for ipv6.level3.com returns "HTTP/1.1 500 Internal Server Error")

Tue Mar 20 20:45:55 UTC 2012

On 03/20/2012 09:54 AM, Jeroen Massar wrote:
> On 2012-03-20 15:40 , Vinny_Abello at Dell.com wrote:
>
> For everybody who is "monitoring" other people's websites, please please
> please, monitor something static like /robots.txt as that can be
> statically served and is kinda appropriate as it is intended for robots.

This could provide a false positive if one is interested in ensuring
that the full application stack is working.

> Oh and of course do set the User-Agent to something logical and to be
> super nice include a contact address so that people who do check their
> logs once in a while for fishy things they at least know what is
> happening there and that it is not a process run afoul or something.

A server side process? Or client side? If the client side monitoring is
too aggressive , then your rate limiting firewall rules should kick in
and block it. If you don't have a rate limiting firewall on your web
server, (on the server itself, not in front of it) then you have bigger
problems.

> Of course, asking before doing tends to be a good idea too.

If you are running a public service, expect it to get
monitored/attacked/probed etc. If you don't want traffic from certain
sources then block it.

> The IPv6 Internet already consists way too much out of monitoring by
> pulling pages and doing pings...

Who made you the arbiter of acceptable automated traffic levels?

>
>
>  (who noticed a certain s....h company performing latency checks against
> one of his sites, which was no problem, but the fact that they where
> causing almost more hits/traffic/load than normal clients was a bit on
> the much side,

Again. Use a firewall and limit them if the traffic isn't in line with
your site policies.

>  And for the few folks putting nagios's on other people's sites, they
> obviously do not understand that even if the alarm goes off that
> something is broken that they cannot fix it anyway, thus why bother...

You obviously do not understand why people are implementing these
monitors. It's to serve as a canary for v6 connectivity issues. If I was
implementing a monitor like this, I'd use the following logic:

HTTP 200 returned via v4/v6 == all is well
HTTP 200 returned via v4 or v6 , no HTTP code returned via v4 or v6 (ie
one path works) ==  v6/v4 potentially broken.
no HTTP code returned via either method == end site problem. nothing we
can do. don't alert.

Presumably you'd also implement a TCP 80 check as well.