[announce-all] NAC Maintenance this weekend (fwd)
alex at nac.net
Fri Dec 10 18:18:33 UTC 1999
Something interesting for all you operational folks... got this from a
---------- Forwarded message ----------
Date: Fri, 10 Dec 1999 05:11:42 -0500
From: Protect_the_innocent <nobody at nowhere>
To: 'Alex Rubenstein' <alex at nac.net>
Subject: RE: [announce-all] NAC Maintenance this weekend
We tested our Y2k power contingency plans just over two weeks ago... we
didn't expect any downtime either. We moved up the date on our power
systems and cut main power.
Three generators failed to operate. Two threw their breakers due to the high
load on them. The last was overwhelmed when the previous two cut out and
threw an estimated 300 amp surge through the lines, blowing up hundreds of
light bulbs, frying fax machines and pencil sharpeners along the way,
jumping circuits in our industrial level surge protector and traveling
through our "surge protected" lines to every desktop in the company and into
the server room. After taking out over 100 monitors and almost 40 PCs, the
surge proceeded to destroy our Server Room air conditioner, 4 huge APC UPS
systems, 4 Compaq Proliant 6500s, 9 Compaq Proliant 3000s, a Compaq Proliant
1800 and both AS/400s.
The surge then jumped the lines into the main power grid, blowing up two
transformers, one of which fell on the IT manager's car (poetic justice) and
cutting power on the entire block. The generator then proceeded to burst
into flames, eventually blowing up all six generators and burning up seven
This all occurred within about 10 seconds and sent 38 people to the
hospital, got the entire IT staff fired (and me hired as the new IT
manager,) cost the company over $650,000 in equipment (not to mention the
impending lawsuits), destroyed 8 cars and has cost the company two weeks
(and running) of downtime.
The resulting publicity got us on television in 5 states and caused our main
competitor to announce that they will be closed for the week following New
Let's just say I'll make other plans for internet access for tonight and the
weekend - just in case.
From: Alex Rubenstein [mailto:alex at nac.net]
Sent: Thursday, December 09, 1999 10:43 PM
To: announce-all at nac.net; operations at nac.net
Cc: network at nac.net; internal at nac.net
Subject: [announce-all] NAC Maintenance this weekend
** This electronic communication is being originated from Net Access
** Corporation, your Internet Service Provider.
In an ongoing effort to make our systems ready for the Year 2000 date
rollover, we will be performing some tests and upgrades this weekend.
On December 10th, 1999, at 5:00pm EST, we will be testing our Year 2000
power contingency plan. No downtime is expected during this test.
On December 11th, 1999, as 2:00am EST, we will upgrading william.nac.net,
a virtual web server. Expected downtime is four hours. Customers who have
websites on this virtual web server will be out of service during this
If you have any questions, please call the Technical Assistance Centre at
More information about the NANOG