outages list down? Lightower, Worcester, MA fiber cut

Fri Nov 7 13:43:57 UTC 2014

RFO: underlying filesystem issues on the hypervisor caused the system to “freak out” (yes, that’s the technical term).  After a reboot, some fighting with java iKVM the system is back up.  Sorry for the trouble.  

OT: (you can delete if you don’t care about my random stuff)

I have some bigger plans to start to split the various parts of puck into more distributed systems (e.g.: DNS services, mail lists, etc) of time, including the various open.*project sites and databases).

Last hardware refresh was in fall of 2011, so with the falling prices of SSD, etc it’s likely time to transition.

- Jared

> On Nov 7, 2014, at 7:34 AM, Chuck Anderson <cra at WPI.EDU> wrote:
> 
> I can't get to the mailing list page, and neither can
> downforeveryoneorjustme.com:
> 
> http://downforeveryoneorjustme.com/puck.nether.net
> 
> I've sent a couple new email messages there and they haven't been
> delivered either.
> 
> Here is the latest message I sent regarding the Lightower, Worcester,
> MA fiber cut:
> 
> Date: Fri, 7 Nov 2014 07:26:55 -0500
> From: Chuck Anderson <cra at WPI.EDU>
> To: outages at outages.org
> Subject: Re: [outages] Lightower, Worcester, MA fiber cut
> 
> Update as of 06:05:16 -0500:
> 
> "Status has remained the same Lightower has two (2) 288 cables that
> were burnt through in a National Grid  enclosure. National Grid has
> informed us that there is an exposed high tension power line in this
> enclosure and there will be no access granted in the near
> future. National Grid has deemed this manhole unsafe at this
> time. Lightower is working on an alternate solution that will re-route
> fibers around the affected man hole still NO ETTR at this time. We
> apologize for this inconvenience."
> 
> On Fri, Nov 07, 2014 at 04:04:27AM -0500, Chuck Anderson wrote:
>> Updates as of 01:41:40 -0500 and 02:58:11 -0500:
>> 
>> "We have two (2) 288 cables that were burnt through in this National
>> Grid enclosure. National Grid has informed us that there is an exposed
>> high tension power line in this enclosure and there will no access
>> granted in the near future. National Grid has deemed this manhole
>> unsafe at this time. Lightower is working on an alternate solution
>> that will re-route fibers around the affected man hole. NO ETTR at
>> this time."
>> 
>> My initial outage time was exactly 18:58:54 EST on two circuits.
>> 
>> On Thu, Nov 06, 2014 at 08:49:16PM -0500, Chuck Anderson via Outages wrote:
>>> We lost several dark fiber circuits out of Lightower in Worcester, MA.
>>> There is a manhole fire down the street at Main St. & Highland
>>> St. that is most likely the cause.  Smoke is billowing out of several
>>> manholes.  NGrid, NStar, police & fire are on the scene.  Lightower
>>> has dispatched techs.  No ETR yet.
>>> 
>>> One other of my circuits that leaves the building via a diverse path
>>> is still up luckily, or we'd be off the net entirely.