Nashville

Thu Jan 14 15:35:20 UTC 2021

I wouldn't say bad design, I would say outdated design. How do you take a
single fiber optic cable or a copper cable bundle going to a
business/building or a house and terminate it at 2 different Central
Offices? It inherently has a single points of failure. (why I added extra
straps to my fiber outside at home because stupid wind blows the damn cable
around that goes to the pole, I'm in NJ everything on poles here)

I don't think a bomb going off was part of the redundancy design process
when telephone central offices were first starting to be built.
I heard the rumour is (perhaps fact) the generators were keeping everything
up till about 12-1 PM eastern when they shut the gas supply off to the
neighborhood earlier safety. Once the batteries drained, that was it.

Family I have in Nashville were on the phone when it cut out and then
everything went dark.

I do think they could have done a better job with the wireless infra.

No reason you can't have some cell sites that can feed to a different
central office.

The fact is, distributed systems could work better, many smaller switching
and distribution buildings scattered about a metro area with redundant
links. A mesh. That was the single point of failure is just the link from
your house to the next hop. Obviously not feasible with copper pairs.

- Javier

On Thu, Jan 14, 2021 at 10:05 AM Hiers, David <David.Hiers at cdk.com> wrote:

> No doubt they’re good, but the best support can’t overcome bad design.
>
>
>
>
>
>
>
> *From:* NANOG <nanog-bounces+david.hiers=cdk.com at nanog.org> *On Behalf Of
> *Robert DeVita
> *Sent:* Tuesday, December 29, 2020 5:17 PM
> *To:* Eric Kuhnke <eric.kuhnke at gmail.com>; Sean Donelan <sean at donelan.com>
> *Cc:* NANOG <nanog at nanog.org>
> *Subject:* Re: Nashville
>
>
>
> AT&T Disaster Recovery Team is probably the best in the business. The
> resources they can bring to the table are unmatched. This would have been
> 100x worse if it hit a carrier neutral datacenter. They don’t have nearly
> the same resources to restore something like this. They usually do a road
> show (pre Covid). If you get a chance it’s definitely something you should
> go check out. Very impressive.
>
>
>
> Robert DeVita
>
> Founder & CEO
>
> Mejeticks
>
> c. 469-441-8864
>
> e. radevita at mejeticks.com
> ------------------------------
>
> *From:* NANOG <nanog-bounces+radevita=mejeticks.com at nanog.org> on behalf
> of Eric Kuhnke <eric.kuhnke at gmail.com>
> *Sent:* Tuesday, December 29, 2020 5:06:00 PM
> *To:* Sean Donelan <sean at donelan.com>
> *Cc:* NANOG <nanog at nanog.org>
> *Subject:* Re: Nashville
>
>
>
> From a few days ago. Obviously centralizing lots of ss7/pstn stuff all in
> one place has a long recovery time when it's physically damaged. Something
> to think about for entities that own and operate traditional telco COs and
> their plans for disaster recovery.
>
>
>
>
>
> Nv1
>
>
>
> Here is the latest update:  6:46AM 12/27:
>
>
>
> Work continues restoring service to the CRS routers in the Nashville
> Central Office. One router remains out of service and the other is in
> service with some links remaining out of service.
>
>
>
> The working bridge will reconvene at 08:00 CT with the following action
> plan:
>
> Additional cabling added to the first portable generator to enable full
> load capabilities (08:00 CT)
>
> Pigtails with camlocks installed for easy swap; investigate possibility to
> land generator on the emergency service board to give the site N+1 with a
> manual ability to choose anyone. (08:00 CT)
>
> check small power plants on floors 4 and 6 (08:00 CT)
>
> Investigate water damage on 1st floor and energize if safe (08:00 CT
>
> Air handlers for floors 4,5 and 6 (09:00 CT)
>
> complete all transport work
>
> Turn up SS7
>
> Turn up 911 service - Approximately noon or after)
>
> Turn up switching service.
>
> TDM Switching team will reconvene at 09:00 CT and the Signaling team will
> reconvene at 11:00 CT on 12/27/2020.
>
> DMS equipment on the 1st floor will be assessed for water damage.
> Switching teams will monitor power and HVAC restoration and will begin
> switch restoration as soon as the go ahead is provided by the power team.
>
>
>
> Recovery Priorities:
>
> 1. 4th & 5th floors (Specify transport equipment needed to clear MTSO SS7
> isolation & Datakit needed for Local Switch restoration). Transport SMEs
> currently working to turn up transport equipment
>
> 2. 6th floor (ESINET Groomers)
>
> 3. 10th and 8th floors (N4E) – Trunks
>
> 4. 1st floor (DMS: DS1, 5E: DS3) - Local POTS
>
> 5. 1st floor (DMS: DS0, DS2 | 5E: DS6) – Trunks
>
> 6. 11th floor (DMS: 01T) – Trunks
>
> 7. 4th floor (STP and SCP with mates up in Donelson)
>
>
>
> The next update will be issued at approximately 09:00 CT on December 27.
>
>
>
>
>
>
>
> Nv2
>
>
>
> As of 09:00 CT: Teams worked through the night to restore service and
> improve conditions at the Nashville 2nd Ave Central Office. Since the
> initial service impact, over 75% of the Out of Service Mobility Sites have
> been restored. Certain call flows may be limited and should improve as
> additional restoration activities complete.
>
> The generator that is currently powering equipment on the 2nd and 3rd
> floor, was refueled and ran with no issues through the night. Overnight,
> the batteries connected to it, continued to charge. Teams have placed
> additional power cables, which once connected, will allow the working
> generator, to better handle the load in the building. In order to
> accomplish this, the generator will need to be shut down for 15-30 minutes
> this morning, so teams can connect the new cables to the system. The power
> team reports they are still on target to restore power and cooling to the
> 5th and 6th floor by approximately 12:00 CT. Also, a portable chiller will
> be delivered this morning and strategically placed, in case it is needed to
> assist in cooling the office.
>
> There is a Call Center at 333 Commerce, in Nashville that does not have
> network or phone services available. Corporate Real Estate (CRE) reports
> there is some damage to that office, but the extent of the damage will not
> be known until they can gain access to the site. Because of this, the
> impacted Call Center ceased operations until further notice.
>
> DMS switching equipment on the 1st floor will be assessed for water
> damage. Switching teams will monitor power and HVAC restoration. Equipment
> power ups will begin, as soon as the go ahead is provided by the power
> team.
>
> Two SatCOLTs remain positioned on the East and West sides of the NSVLTNMT
> Central Office providing critical communication for teams working
> restoration efforts. There are 17 assets deployed in the field- 15 are on
> air (the 2 at the CO and 13 supporting FN Customer Requests) and 2 are in
> hot-standby for FN Customers where macro service recently recovered. There
> is 1 asset staged at a deployment site in KY where macro service restored,
> and 8 additional assets are on route to Nashville today to fulfill pending
> FN Customer requests. Incoming requests continue to be triaged. The ones in
> areas where service looks to have been restored, are being held, while the
> others are being prioritized to be dispatched upon.
>
>
>
> The next update will be issued at approximately 14:00 CT, unless there is
> a significant change in status.
>
>
>
>
>
>
>
> Nv3
>
>
>
> AT&T Nashville update below, received at 3:35PM 12/27.
>
>
>
> Since the initial service impact, over 95% of the Out of Service Mobility
> Sites have been restored. Certain call flows may be limited and should
> improve as additional restoration activities complete.
>
>
>
> Electricians have installed the additional power cables from the
> generator, to the emergency bus. These new cables will allow the generator
> to support more of the load, of the building. The portable chiller
> requested, has arrived on-site, and is available to assist in cooling, if
> needed. Generally speaking, there are four (4) phases of restoration per
> floor (Air Handler restoral, Power restoral, Transport Equipment restoral,
> and Switch/Application Equipment restoral). Teams report that Air Handlers
> are up and running, and all power plants are on floors 2 through 7 are
> online. Given significant progress made, floors 2 through 7, are ready for
> technology turn up. Relative to Priority Transport related equipment,
> approximately 90% of the elements have been turned up on floors 2 through
> 7. The Power team is currently working on Floors 8 through 11 (N4E). The
> first floor is not accessible, at this time. Once access is granted by
> federal and local authorities, further assessment and restoration efforts
> will begin.
>
>
>
> The generator is currently supporting approximately 50% of its capacity,
> and alternative plans are being considered to handle the full load of the
> building. Teams continue to work proactively in effort to identify
> potential issues and are actively engaged working to restore services and
> repair infrastructure.
>
>
>
> AT&T Network Disaster Recovery (NDR) has eleven (11) SatCOLTs in service
> (TN, AL, GA). Two (2) of the eleven (11) are deployed at the Nashville, TN
> Central Office to provide coverage for the AT&T response teams as well as
> FirstNet (FN) customers. One (1) COLT is in hot-standby (TN). Six (6) COLTs
> are en-route to deployment sites in TN and AL. Three (3) COLTs are being
> demobilized in Alabama and coming back to Nashville for new assignments and
> five (5) additional COLTs are en-route to the Nashville area to support
> additional requests.
>
>
>
> The next update will be issued at approximately 19:00 CT, unless there is
> a significant change in status.
>
>
>
>
>
>
>
> On Mon, Dec 28, 2020, 5:59 PM Sean Donelan < sean at donelan.com > wrote:
>
>
> AT&T statement says nearly all services have been restore in Nashville as
> of Monday, 5pm CST
>
> They are working on permanent repairs.
>
> https://about.att.com/pages/disaster_relief/nashville.html
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mila.bitdam.com_api_v1.0_links_rewrite-5Fclick_-3Frewrite-5Ftoken-3DeyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZiOGZhNWRhYjg5ZjE0NzJiIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.fQSGLDVfXTrVFYepbx0AR-5FuG-5FqKj-2DUG337RhC567WBI-26url-3Dhttps-253A__about.att.com_pages_disaster-5Frelief_nashville.html&d=DwMF-g&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=QP--yQuEX4jG-74vYtzEjEKuDLrRan890jWR_pl9hjU&s=kRIGqeCVD61v63ETdC8ooxHNNAP0FfZyzYnYzgnpLPQ&e=>
>
>
> AT&T's Network Disaster Recovery group faces management questions nearly
> every year to justifying their budget. While no one wants disasters,
> business continuity has to be part of the business.  There are also mutual
> aid agreements between companies, but I don't know how many were invoked
> for this incident.
>
> https://about.att.com/ecms/dam/pages/disaster_relief/NDR_edited_04.22.19.pdf
>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__mila.bitdam.com_api_v1.0_links_rewrite-5Fclick_-3Frewrite-5Ftoken-3DeyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZXdyaXRlX2lkIjoiNWZlYmI3MGZlZTIwYTJmMzM5ZGM5ZTgwIiwidXJsIjoiIiwib3JnYW5pemF0aW9uX2lkIjoxNzgxfQ.xIE6w200hxqs5MLmTyAw3TKHl50TG9NaxjGW32JTEr8-26url-3Dhttps-253A__about.att.com_ecms_dam_pages_disaster-5Frelief_NDR-5Fedited-5F04.22.19.pdf&d=DwMF-g&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=QP--yQuEX4jG-74vYtzEjEKuDLrRan890jWR_pl9hjU&s=oSnRWvbMLFwj9B8SlCbqsU06WNkyqf2fALfQ2mlcMJg&e=>
>
> ------------------------------
>
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20210114/678bc5d5/attachment.html>