Katrina: directNIC Stays Online - Blog + Images

Wed Aug 31 14:22:51 UTC 2005

On Wed, 31 Aug 2005 15:47:43 +0200
 Chris Gilbert <Chris at LainOS.org> wrote:
> 
> For those that don't know, directNIC as well as some other NOCs are
> located together in a high-rise in New Orleans.
> 
> Despite everyone's warnings (including my own!) some tough-as-nails guys
> from directNIC stayed behind to battle the hurricane and keep the
> networks online.
> 
> [snip]
> While the safety of our staff is paramount and most were safely
> evacuated, a small group of key personnel stayed behind to safeguard our
> data center and make sure that all of our services remained online and
> stable. During this time, while we spent a great deal of effort battling
> broken windows, incoming water, and flying debris, our hosting and
> registration services remained online and worked flawlessly.
> [/snip]
> 
> Link: http://www.directnic.com/katrina.php (Pictures too!)
> 
> It also mentions the power shortage, and that they are using diesel
> generated power.
> 
> I remember a week or two ago people were talking about building
> redundant datacenters, off-grid power, failure mitigation, etc.
> 

I remember that after 9/11 the real network hits started about 3 days later, when
the diesel generators started running out of fuel  in the downtown telco hotels, and there
was  no  way to physically get fuel trucks to their location. The  equipment and generators were
fine, they just ran out of fuel.

If you look at the flooding  in downtown New Orleans, it looks like this might happen again there.

It makes me wonder whether part of disaster planning shouldn't be  some sort of power  triage, where
if it looks like  it's not going to be possible to get fuel  to a datacenter after a systemwide
power outage, instead of powering everything for a short time and then going dark, a subset is
powered for weeks.

Since I believe that air conditioning is a big  part of the fuel expenditure, this  might imply
preplanning to the extent of grouping essential  equipment  together in a limited area that
could  be kept cool when  everything else went dark.

> I think if nothing else, this is at least a success story of building a
> NOC which can provide critical infrastructure that will survive major
> disasters.
> 
> --
> Regards,
> Chris Gilbert

Regards
Marshall  Eubanks