Completewhois New Features - RBL Lookup and Search Utilities

Tue Jul 26 19:27:43 UTC 2005

Hello everyone,

Over the past month several new features and utilities have been added
that are likely to be of interest here. In this post I'll focus on RBL 
lookup related utilities which have to do with rbl data from about 30 
lists (with one or two exceptions, pretty much covers 25 most used
free lists) that we're collecting and aggregating in the database 
(updates once per day) for analysis and allow users to check on based
on individual queries and ip ranges.

I. First of there is now direct whois lookup facility to check if ip 
address or domain is in one of those lists - you can simply do
  whois -h rbl.completewhois.com ip-address/domain
OR (to show all lists that were checked)
  whois -h rbl.completewhois.com RBL_INCLUDENOMATCH=ON ip-address/domain
OR (to include RBL data with normal whois lookup)
  whois -h whois.completewhois.com RBL ip-address/domain

The queries are completed in average 1/2 second, so results are always fast.
Note that no ip ranges are accepted (or going to be) on the whois interface.

II. The web interface and light documentation for RBL Lookup (individual
queries interface to our system) is available at
  http://www.completewhois.com/rbl_lookup.htm

The website utility has two types of output display - one user-friendly 
(now default) table showing lists that matched and did not as red and 
green and including links to the list pages (good for less RBL-familiar
users who want to know what to do) and simple format based on whois (can 
be easy to cut-paste from) and which is used when you want to also combine
query with whois and dns data.

The website lookup CGI can also be refernced directly (already is and used 
quite heavily) from other places and applications, do it as in this example:
  http://www.completewhois.com/cgi-bin/rbl_lookup.cgi?query=62.139.100.213

There is also real-time RBL check utility for 200 lists (not using our 
database and so quite slow) available on the bottom of the page and you
even have a choice there to use several dns libraries (ADNS, FireDNS, BIND
resolver) and compare how fast/slow they work...

III. Another utility on the website allows to do IP range searches and
is intended to be used primarily by ISPs and network operators to check
on  the listings that cover their own ip blocks (this is to help make 
operators aware of the extent of possible abuse coming from their network).

The interface to this is available at:
  http://www.completewhois.com/rbl_search.htm

The search utility is restricted to maximum /24 range as allowing more
then that could in my opinion (and others I consulted) facilitate abuse
rather then help stop it. To be able to do more then /24 lookup on your
ip block(s), you will need to register and get username and password in
our system (its still all free - registration is just making sure only
ISP who is assigned the ip block can do query on entire block). Also note
that use of this utility is covered by separate AUP.

The results from ip range search also come in several formats, including
simple list on the website, comprehensive webtable format as well as an
option to produce CSV file for export to spreadsheet. The queries and 
searches are typically done in 1-2 seconds for /24 and about 4-8 seconds 
for /16 (with 1000 matches from various RBLs).Webtable adds additional 
couple seconds for large (500+) matches.

In the future if there is an interest, further work will be done on the
search interface ISP features to allow not only one-time lookups but 
reports that can be generated and sent automatically. Options will also 
include ability to do query based on specified time range (i.e. only new 
RBL entries that appeared in last 7 days). You will need to tell me what 
you want and how its to be presented, if you expect new features and note 
that any further work on this will be done end August or later when I come
back from IETF conference.

P.S. For those who like statistics, there are currently 1.8 million
individual RBL entries which as an aggregate cover about 2 /8s (not
all fair comparison because spews level2 covers large ip ranges where
as many other lists are more specific). About 100 thousand (varies,
very low on weekend but can be lot more some days) get updated every
day and most active as far as updates is Spamhaus XBL.

---
William Leibzon
Elan Networks
william at elan.net