NJ impact

Wed Oct 31 19:24:50 UTC 2012

I had to summarize this recently for a news article I was interviewed for, so I figured I forward:

--

Of our three datacenters, this is what we saw:

Parsippany 1 (OCT) - The worst we saw here was several sub-second power hits. UPS's held without problem, and we did not transfer to generator at all yet.

Parsippany 2 (WBR) - Transferred to generator at about 7:55 PM EST Monday as a precautionary measure due to ongoing utility power hits. However, shortly after transfer, utility voltage went to 0 on all phases; around 10p power returned, but abnormally high (seeing about 550 volts on 480 volt bus). We retransferred last night as utility voltage settled down.

Cedar Knolls 1 (MMU) - Briefly transferred to generator around 7:10, then back to utility. We then force transferred to generator around 8pm and stayed until this morning. Returned to utility and all systems are normal.

We have accommodated many customers (and non-customers) by allowing them to use our conference rooms and some offices, mainly because they lost power at their buildings, lost access to their buildings, or generators ran out of fuel. We literally have card tables setup in hallways and common areas where people sit and use Wifi to conduct business.

There has been no service interruption of any sort to any of our datacenter customers, as of this writing.

We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR), network healed in less than 1 second. It was imperceptible. That was early though, Monday afternoon. This has since restored today.

We have sales people onsite who are actively taking sales calls, and there is a measurable amount of customers moving equipment into our facilities as I write this. Many are moving equipment from offices with no power and no ETR in sight. We have electricians onsite to help accommodate.

The facilities department has been manning all locations around the clock.

All sites remain accessible by road. There was never a situation where you could not over the last 48 hours.

We have no equipment in 111 8th, so I cannot offer any telemetry there. However, our gear in 60 Hudson St (Tel-X 9th floor) is still online.

We saw no interruption at 165 Halsey, except for some very brief utility hits. UPS's held.

We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet customers down - presumably mostly power loss on the CPE. We lost our connection to Covad, presumably they have gear in a building which lost power. These are only just starting to come back.

We have seen substantial interruption to the cellular networks; in many cases, ATT Wireless (who NAC uses) is unable to complete a call. In certain areas, there is no signal at all.

Where I live, in 07874 (Byram), many roads are impassable and most everyone does not have power. Trees are down everywhere. I heard that a Byram squad car had a tree land on it while responding to a call. Schools are closed, Halloween has been "rescheduled" according to our governor. My cable is down, cable modem is down, POTS service is out - yet I still have xDSL (loop by Verizon, IP by NAC) and I run on that successfully. I am unable to get any ATT cell service at my house (usually at least three bars). I am on generator at my house, consuming about 2 gal/hour (propane), with a little over 500 gallons left.

Gasonline is becoming an issue. As I understand, having talked to our oil supplier, there is 0 refinery capacity in NJ at this time. All gasoline is being imported from PA, DE, MD, etc. Most stations don't have power. Ones that do are generally out of gas. If they have gas, they have mile long lines and are rationing.

Diesel is unavailable if you had not pre-arranged for it.

Widespread power outages remain. I am running on generator at my house (Generac QT048) on a 1000 gallon tank of propane, have been since Monday about 7pm. It is estimated at least 7 to 10 days before power restoration. I have yet to see one JCPL truck anywhere near where I live.

That is all I have.

This is the road leading to the Byram Police Dept - I took this yesterday morning:
https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n.jpg

This is typically what Northern NJ looks like:

https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048_n.jpg

https://sphotos-b.xx.fbcdn.net/hphotos-ash3/61115_4317167440308_289655722_n.jpg

(not the top of the telephone pole hanging from the wires)