69/8...this sucks -- Centralizing filtering..

Fri Mar 14 16:53:12 UTC 2003

[ apologies for the long post ]

On 2003-03-11 19:57:04 +0000, jlewis at lewis.org wrote:
> 
> Also, on a side rant here....Why do all the RIR's have to give out
> whois data in different, incompatible, referal-breaking formats?

The reason for the different formats is partly historical, and
partially a result of the fundamental differences between the RIR's.

The historical reason is that each RIR has a different origin, and the
databases and Whois software comes from that origin.  The RIPE NCC
started with nothing, evolved to RIPE-181, then RPSL, and is now
moving to RPSLng + extensions.  APNIC adopted RIPE NCC software, and
is very nearly compatible.  ARIN's database was inherited from the
InterNIC, and has since evolved into a new, organisation-based model.
I believe LACNIC's database is inherited from the Brazil domain name
registry, so it uses that format (this is the one I am least familiar
with - so I may be in error).

The formats remain different because the RIR's have evolved their
databases to solve problems that are most important in their regions.
For instance, ARIN has chosen a model that reflects registration in a
straightforward way, whereas RPSL is useful for operators wanting to
document policy.

> The next step in my work once my ping sweep is complete (looks like
> that'll be today) is going to be to take a list of what looks like
> it'll be ~1000 IPs and generate a list of the unique networks that
> are broken.  To do this, it'd be nice if there were some key I could
> get from whois, store in a column, select a unique set from, then
> reuse to lookup POCs from whois, and send off the emails.
> 
> registro.br and LACNIC entries start with inetnum: using what I'll
> call brief CIDR, i.e.
> inetnum:  200.198.128/19
> 
> APNIC and RIPE entries start with inetnum:, but use range format.
> i.e.
> inetnum:      203.145.160.0 - 203.145.191.255
> 
> ARIN entries include fields like
> NetRange:   128.63.0.0 - 128.63.255.255 
> CIDR:       128.63.0.0/16 
> 
> The APNIC and RIPE NetRange/inetnum fields are easy enough to deal
> with, but send a whois request for 200.198.128/19 to whois.arin.net
> and you get "No match found".  Send it as 200.198.128, and
> whois.arin.net will refer you to whois.lacnic.net.  Send it to
> whois.lacnic.net as 200.198.128, and you get "Invalid IP or CIDR
> block".
> 
> I realize programming around all this is by no means an
> insurmountable task, but it is a pain.  It'd be ideal if there were
> a unique key field, say Net-ID included in the whois output from all
> the RIR whois servers that could be used to identify the network and
> the appropriate whois server.  i.e.
> 
> NetID: 200.198.128.0 at whois.lacnic.net

In the current situation, users must query up to 4 servers
(whois.apnic.net, whois.arin.net, whois.lacnic.net, and
whois.ripe.net) to find information about an IP address, in some cases
without any way of knowing which RIR has allocated the space.  Each
RIR parses queries and presents results in a different format.

This is not ideal - to put it mildly.

The good news is that we are aware of the problem, and not sitting on
our laurels.  The eventual goal is to answer a query for IP or AS
space at each RIR, using the "native" query and result format, and get
the best possible answer.  We've completed part of the mapping between 
schemas, and after that is finished it's just a matter of writing some
software.  ;)

There is also a technology that might come out of the CRISP IETF
working group:

http://www.ietf.org/html.charters/crisp-charter.html

We (the RIR's) are tracking this work.  Since this involves an actual
protocol difference from our beloved Whois protocol, if it is adopted
it will certainly take longer to adopt.  But there is no reason the
two protocols can't co-exist and complement each other.

If you have any interest in participating in RIPE Database-related
issues, please feel free to join the mailing list:

http://www.ripe.net/ripe/wg/db/index.html

We (meaning the RIPE NCC, especially the database group) take a lot of
our direction from the DB working group.  It's open to all.

-- 
Shane Kerr
Database Group Manager
RIPE NCC