Bogon stupidity... warning... operational post.

william(at)elan.net william at elan.net
Thu Dec 22 19:00:51 UTC 2005



On Thu, 22 Dec 2005, Christopher L. Morrow wrote:

>> Yes, legacy blocks (with large number of smaller allocations) whenever
>> datasize during processing exceeded certain amount. The bad data was
>> present at 2 of 4 servers for duration of the night but dns was being
>
> so 50+% of your system was hozed for some long period of time :( bad.

Yes, bad reaction time - when problems were reported last year, it was 
dealt with in < 2 hours. It is always bad when things happen during the
night, but as engineers we typically plan updates for that time thinking
it would have less impact on everyone - sometimes it backfires.

>> changes same time as well, so I don't know how much affect there was
>> but apparently considerable; this is the most serious problem in months.
>
> 'most serious problem in months' ... this has happened in smaller chunks
> during the past 'months' ? yikes... is that noted on your site so users of
> the 'service' will know what sorts of 'problems' they might be
> encountering due to their reliance on this 'service'?

I knew the system was out of space and that is why the change to new 
system, although in last couple days I had to rush a bit to finish it.
Previously the errors were confined to comparisons to what is in bgp, 
so mostly not operationally important except for some research
(also in couple cases last months impacted ip->country data files
causing some country data to not list any ip blocks).

This is not typical of the service and as I already noted. New server
has 8% of space used right now (enough for 5+ years if nothing seriously
changes; plus access to storage server with 1.5TB space) which means the 
same should not repeat. However I still have days of work to finish this
upgrade in other ways.

-- 
William Leibzon
Elan Networks
william at elan.net



More information about the NANOG mailing list