75th Broad up/down?

Wed Oct 31 19:14:05 UTC 2012

Here's a few notes from Internap which has stuff at 75 Broad and 111 8th:

---- Forwarded Message ----

Hurricane Update: Power Restored at LGA9, 111 8th Ave.

Update:

We have restored all power to the LGA9 site via generator.  Phase 1&2
UPS UPS modules are in bypass and being investigated at this time.  We
will be moving the UPS units back inline one at a time as they are
verified over the coming hours.  There is no impact to customer load
expected.  Phase 3 load is currently on UPS conditioned power supported by
generator.

All INAP IP service points are operating normally.  Our staff is currently
working to bring LGA9 Agile customers back up.

---- Forwarded Message ----

At approximately 23:20 local time, the LGA11 facility returned to
service via generator power, including critical customers loads. The
generators were refuelled and restarted and are currently operating
normally, carrying the critical load. Additionally UPS systems have since
been brought back online and are now helping to support power onsite.
Additional fuel supply is being arranged and there is ample fuel on site
at the moment to maintain operation.

As noted, power to customer gear has been restored, however building
access is still restricted and customers are not yet able to visit the
site.

---- Forwarded Message ----

At approximately 05:45 EDT connectivity to Internap's LGA11 datacenter was lost.  This loss was caused by 2 other datacenter power failures that provide fiber to LGA11. We are working with the building personnel at both sites to correct the problems. We will provide more updates as information becomes available.

---- Forwarded Message ----

At approximately 23:20 local time, Internap's LGA11 facility returned to service via generator power, including critical customers loads. The generators were refueled and restarted and are currently operating normally, carrying the critical load. Additionally UPS systems have since been brought back online and are now helping to support power onsite. There is ample fuel on site at the moment to maintain operation and there is a scheduled delivery of additional fuel. This will continue until Con Ed restores grid power.  Most all servers are not back online and accessible.  Please log in and confirm that everything is working as expected in your environment.  

> An online magazine I work with is, we believe, hosted at 75 Broad or 111 8th. It dropped out Monday night, right after they announced that the diesel pumps had fried due to flooding at 75 Broad, came back up early Tuesday morning, and then died again this morning.
> 
> IP address is in 74.63.44.0/24 block (which may be part of a larger block).
> 
> Pinging 74.63.44.255 gets me replies from 208.122.44.210 only.
> 
> Anyone know the current status of these facilities? I've got an e-mail in to noc at internap but haven't heard back - and if they have a facility down they're probably too busy to reply to a single web hosting customer.
> 
> I have seen the report from earlier this morning indicating that 111 8th had an issue that's been fixed - but the Internap NOC line (877.843.4662) still indicates that NYM008 is having a power outage.
> 
> Thanks!
> 
> -- 
> Jeff Shultz
> 
> 
> 
> 