Distributed DNS/etc checking

Mon Apr 2 16:08:06 UTC 2012

Good day all,

There have been a few instances where we've wanted to check our external
DNS servers from various external networks, so we've utilized the existing
looking glass tools provided by many of you.  However, it's a very manual
process, given that all LG's I've found say no automating/scripting.  If we
want to check from a couple dozen sites around the world, it's a lot of
clicking and typing and collecting.  If we wanted to create an tool that
our NOC could use to verify our services, we would need something we could
script.  Ideally, we'd be able to run this constantly to do health checks
on our services, but one step at a time.

I've been googling, but so far I'm unable to find any larger scale
projects/toolsets that we could use to simplify this process.  Is anyone
aware of something that would allow for me to submit a "job" to some sort
of distributed service (I care about DNS, but others may care about
traceroutes, pings, bgp information, etc), that will then run run the "job"
and give me back an answer?

Similarly, but perhaps differently, those of you who may run large anycast
DNS services, how do you gather "external" stats about routing, response
time, availability, and so on?  It seems like this sort of thing would be a
fairly common requirement (lets see how my network looks to those outside
of it) but everything I can find is very manual at this point.

This looks like a somewhat promising option, however I don't think I could
get buy-in to run a node in our network, so it's not on the table for now:
https://ring.nlnog.net/

This same functionality would likely be very helpful internal to large
networks as well.

I would love to know if I'm missing something obvious, or pieces of
something obvious we could work with.  Failing something already existing,
I'd value any information people care to share about how they do this now,
either on or off list.  I can summarize any findings if the community is
interested.

Cheers,

Todd.