Northern California fires and telecomm outages

Wed Oct 18 05:05:39 UTC 2017

Hi Mike,

Thanks for sharing the story in what must be quite a difficult time.

You mention that Level3 also has presence in the county, including
their own independent route.  Did they also suffer an outage during
this latest incident?  If not, then why did their connection not allow
at least some customers to remain online?

Also, as another poster pointed out, would also be interesting to know
why the microwave link was not already established prior to this
incident if the single point of failure was so well known.
(Especially if your contract with AT&T already mentions that you are
supposedly allowed to have microwave at their CO.)

Cheers,
Constantine.

On 17/10/2017, Mike <mike-nanog at tiedyenetworks.com> wrote:
> Hello fellow operators,
>
>
>      The recent northern California fires wiped us out along with many
> other service providers and services including cellular, landline,
> e-911, cable and the like and I feel it's of operational concern the
> fragility of the telecommunications networks that link us together and
> the experiences had during this most recent event. I hope this topic is
> welcome and I apologize in advance for length.
>
>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>
>      I am the founder and owner of an Internet Service Provider business
> in Mendocino County, California and I also operate a CLEC (Competitive
> Local Exchange Carrier, a type of telecommunications company akin to ATT
> or Frontier), also in Mendocino County, California. The recent Northern
> California fires were again a direct and upfront demonstration of the
> potential for unanticipated widespread public disaster and again
> demonstrated how vulnerable we all are. Especially, the rising
> dependence on broadband service, and an expectation by end users that
> these services which are critical to them will continue functioning
> during emergencies as they may have no other access. There was a
> complete and total breakdown in telecommunication services due to fire
> which wiped out a primary telecommunications cable, and with it, e-911,
> land lines, cellular, cable and other telecommunication services. I hope
> that there will be a more complete public investigation and report on
> this latest incident, but in the mean time I would like to just offer
> what I know and to suggest where we could go from here.
>
>      In the early morning of October 9, the Northern California fires
> were in effect and in Redwood Valley California, a fire was burning that
> destroyed a AT&T telecommunications cable that runs between Ukiah and
> Willits. This single cable is the primary carrier of nearly all
> telecommunications services north of Ukiah. Although I cannot comment on
> the totality of the outage due to this specific fire and line cut, it's
> widely reported elsewhere that services were affected all the way up to
> Arcata at least. For Mendocino County, I can report that this
> telecommunication outage resulted in loss of e-911, cellular, landline,
> cable and most all other broadband services, and caused a complete
> outage of our service too. This resulted in a virtual dark age, where
> reliable and up to date information concerning the emergency situation
> was unable to be communicated, people could not find their families or
> know whether they got out in time because nothing was working and no way
> to talk. We were reduced to using paper and posting notes at the Library
> or other emergency shelter locations or other places where people
> gathered in order to communicate. People in these areas also had to use
> cash to buy supplies. Nothing worked.
>
>      We attempted to engage emergency restoration processes on our own
> for our own customers and services. We determined that we could rapidly
> deploy high capacity microwave connecting a remote location where we
> also already operate microwave, to our main point of presence at an ATT
> building in Ukiah, and provide a sufficient level of service to all of
> our customers that would be a huge benefit during this crisis. As a CLEC
> with an existing presence and power arrangement and so forth within this
> facility, it seemed natural that we should be able to go ahead and get a
> simple OK to go do the work. But, we encountered lethargy and
> insensitivity to the widespread outage and emergency conditions we are
> trying to address. We notified ATT of our need and only days later,
> without followup response, it was suggested that we somehow don't have
> the legal authority to install microwave (despite the fact that our 544
> page contract with ATT that allows our presence within their facilities
> in the first place DOES SAY 'microwave' is allowed), and further they
> suggest we'd have to give up all of our existing contractual rights and
> that if it could happen, it would be only with months of delay and
> endless expense and bureaucratic nonsense that is only designed to slow
> down and de-incentivize any such initiative. The equipment we were
> proposing, is small, low power, lightweight, license free gear that
> could easily fit complete in a car and be deployed within hours without
> prior notice or coordination. The electrical and structural requirements
> are next to nothing, and present absolutely no risk to anyone or the
> facility. We could have had service restored and further, could have
> plugged a critical gap by creating an active backup telecommunications
> path which could be used in future emergencies. I am frustrated with the
> lack of cooperation to restore services during this latest incident and
> I think it speaks volumes about the mentality of the incumbent
> providers. They can't get it together, but here I am standing by with
> the solution at least for my own customers and why, they just don't want
> to allow me to do it either.
>
>      Mendocino County (and northern counties also connected on this same
> telecommunication cable) have suffered similar telecommunications
> outages before. Prior to this event, a cable just north of Hopland was
> cut by two individuals thinking they were stealing copper, resulting in
> widespread outages lasting upwards of 14 hours. And prior to that, a car
> crash on the coast hit the one pole where the cable system comes in from
> the east, wiping out all services along the coast for an extended period
> of time. And before that, a fire in redwood valley just north of coyote
> valley casino along the roadway burned out the cable again as in this
> instance. And before that, another fire north of Willits burned the
> cable again. And the time before that, a backhoe south of Hopland dug up
> the cable and cut it, again resulting in far reaching outages affecting
> hundreds of thousands of people.
>
>      These repeated outages are not being taken seriously and nothing in
> substance is being done to fortify the telecommunications network
> against the next disaster, despite the fact that disaster can and will
> occur again. We have repeatedly drawn attention to these vulnerabilities
> but the public perception and awareness of the risks wanes quickly once
> the crisis is over. I think the incumbent providers have had ample
> opportunity to address the underlaying causes of their repeated
> widespread outages, and the empty promises and assurances they have made
> after past prior disasters should be proof that no actual steps will be
> taken, that they either do not truly understand or care about the effect
> that loss of telecommunications services has and the risk it presents to
> the public.
>
>      The basic fundamental problem facing the telecommunications network
> in Mendocino County, is the fact that it is structured without any sort
> of backup communication links. A primary communications cable extends
> north along highway 101 at Hopland and then goes into Ukiah. This cable
> then splits out at Ukiah and goes north up 101 thru Willits, while
> another segment goes west thru Boonville to Mendocino and then up and
> down along the coast. A cut at any of several points results in the
> complete isolation of communities past the cut as what occurred during
> the recent fire. A basic solution to such a situation then, is to have
> additional cables to increase the connections between areas, so that if
> a cable is cut and disrupts communication flowing one way, that the
> communications can flow over another path from another direction. But
> the incumbent providers have failed so far to do anything and are likely
> to continue to ignore this lack of redundancy.
>
>      I believe that there are some realistic solutions available, and
> based on my business experience in Mendocino County, I have developed
> the beginnings of a plan of action. The short of it is, there needs to
> be a redundant network of geographically diverse, very high capacity
> telecommunication links that connect the core of the County, in order to
> overcome the existing problem of there being only 1 connection today. We
> would need new fiber to connect the major cities from alternate
> directions, and we further would need new connectivity to other sources
> of out of region telecommunication services besides AT&T (Such as
> Level3, who has their own independent cable route into the county). It
> would further require service providers and emergency responders to all
> lease connectivity on this redundant system for full effect, the net
> result however, could be a telecommunications system that stands up and
> continues functioning and with emergency first responders as well as the
> general public able to communicate during emergencies.
>
>      Although I am busy with business and ensuring my customers recover
> from this event, I certainly am happy to add additional background or
> technical information if anyone is interested. Going forward, I also
> could also use help in identifying funding sources and other contacts
> who could help with some of the heavy lifting that is going to be needed
> here and so any pointers would certainly be appreciated.
>
> Thank you.
>
>
>
> Mike Ireton
> President
> Your Town Online, Inc DBA WillitsOnline
> Rural Broadband Now, LLC
>