NANOG Digest, Vol 46, Issue 15
Steve Dispensa
dispensa at phonefactor.com
Mon Nov 7 16:41:59 UTC 2011
Level 3 was down in KC, Chi, and San Jose (at least) for us between
about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.
-Steve
> -----Original Message-----
> From: nanog-request at nanog.org [mailto:nanog-request at nanog.org]
> Sent: Monday, November 07, 2011 10:05 AM
> To: nanog at nanog.org
> Subject: NANOG Digest, Vol 46, Issue 15
>
> Send NANOG mailing list submissions to
> nanog at nanog.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mailman.nanog.org/mailman/listinfo/nanog
> or, via email, send a message with subject or body 'help' to
> nanog-request at nanog.org
>
> You can reach the person managing the list at
> nanog-owner at nanog.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of NANOG digest..."
>
>
> Today's Topics:
>
> 1. Re: Time Warner Telecom problems (Jon Lewis)
> 2. Re: Performance Issues - PTR Records (Leigh Porter)
> 3. Re: Time Warner Telecom problems (Ray Van Dolson)
> 4. General Internet Instability (Jared Mauch)
> 5. Re: TATA problems? (Pierre-Yves Maunier)
> 6. Re: TATA problems? (Leigh Porter)
> 7. Re: Time Warner Telecom problems (Joe Greco)
> 8. Re: TATA problems? (Kelly Kane)
> 9. Re: Time Warner Telecom problems (Blake Hudson)
> 10. RE: Time Warner Telecom problems (Thomas York)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST)
> From: Jon Lewis <jlewis at lewis.org>
> To: Peter Pauly <ppauly at gmail.com>
> Cc: nanog at nanog.org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <Pine.LNX.4.61.1111071005540.24418 at soloth.lewis.org>
> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>
> On Mon, 7 Nov 2011, Peter Pauly wrote:
>
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> I noticed just a little while ago that we're having a lot of DNS fail.
> Initial findings were that several of the root-servers we were trying
to
> reach via our TWTelecom link were unreachable after 2 hops into TWT.
>
> 4 64-128-130-233.static.twtelecom.NET (64.128.130.233) 2.399 ms
2.298
> ms 2.338 ms
> 5 mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18) 11.571 ms
> 11.552 ms 9.467 ms
> 6 * * *
> 7 * * *
> 8 * * *
>
> For instance, a.root-servers.net is pingable from a rackspace server,
but
> not from our network (unless I shut off TWT, at which point it is, but
> it's apparently not the same a.root-servers.net instance rackspace
sees).
> I assume this is one of the root-servers being anycast.
>
> Shutting off our BGP with TWT didn't appear to help (though the
> root-servers became reachable)...so I assume there's more going on
than
> just TWT routing fail.
>
> ----------------------------------------------------------------------
> Jon Lewis, MCP :) | I route
> Senior Network Engineer | therefore you are
> Atlantic Net |
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 7 Nov 2011 15:29:30 +0000
> From: Leigh Porter <leigh.porter at ukbroadband.com>
> To: Bj?rn Mork <bjorn at mork.no>
> Cc: "nanog at nanog.org" <nanog at nanog.org>
> Subject: Re: Performance Issues - PTR Records
> Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863 at ukbroadband.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
>
> On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn at mork.no> wrote:
>
> > Leigh Porter <leigh.porter at ukbroadband.com> writes:
> >
> >> Indeed, there is no way I would allow that either. But really,
> >> providing a reverse zone and forward zone to match is a case of
five
> >> minutes and a shell script or a DNS that as Steinar said, will
> >> synthesise results.
> >>
> >> It's really not all that difficult..
> >
> > No, not at all. It's just totally pointless. Any IPv6 address is
just
> > as pretty as a synthesized name. Maybe even prettier. Do you prefer
> > "2001:db8:1::2" or
"20010db8000100000000000000000002.rev.example.com"?
> >
> > If we're going to provide any reverse DNS for end users, then it is
> > because we can create names which actually improves something.
> >
> >
> > Bj?rn
> >
> >
>
> Yup it is pointless.. Mine are all ipadrress.domain which is of
course,
> pointless.. I suppose at least somebody would glean that perhaps its a
> home user rather than a business or server on that address but that's
all.
>
> With IPv6 arguably even more pointless as you say.
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 7 Nov 2011 07:28:18 -0800
> From: Ray Van Dolson <rvandolson at esri.com>
> To: nanog at nanog.org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <20111107152817.GA29715 at esri.com>
> Content-Type: text/plain; charset=us-ascii
>
> On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> FWIW, my home TWC connection dropped this morning for about 15 minutes
> (Southern California around 6:30AM'ish). Still could ping the default
> gateway, but packets weren't traversing much beyond that.
>
> Didn't investigate further, just headed into work.
>
> Ray
>
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 7 Nov 2011 10:31:31 -0500
> From: Jared Mauch <jared at puck.nether.net>
> To: Tom Hill <tom at ninjabadger.net>
> Cc: nanog at nanog.org
> Subject: General Internet Instability
> Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD at puck.nether.net>
> Content-Type: text/plain; charset=us-ascii
>
> On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
>
> > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> >> We seem to be having some problems with our tata links - first seen
in
> EU
> >> about 45 minutes ago, now we're seeing problems in NA. I'm focused
on
> DNS,
> >> so I'm seeing a lot of timeouts/servfails, but our networking folks
are
> >> talking about links dropping.
> >>
> >> Anyone else seeing oddness on the NA Internet right now?
> >>
> >> http://downrightnow.com/ confirms - something is up.
> >
> > There are widespread issues across the Internet; certain versions of
> > Juniper firmware have core dumped after seeing a particular BGP
'UPDATE'
> > message.
> >
> > (That's the running theory at least).
> >
> > It's affected multiple service providers, globally, not just those
> > connected to TATA.
>
>
> Pretty much any major BGP event will impact multiple providers.
>
> A threshold you should use to view the general instability (which I
find
> valuable, you may as well) is route views data.
>
> If you look at the BGP UPDATES archive sizes, you can see when
something
> happens, e.g.:
>
> http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
>
> Take a look at the size of the updates.20111107.1400.bz2 file and the
1415
> file. They are abnormally large compared to a normal period of time.
> This shows there were a lot of updates out there being processed and a
> reference to levels of instability.
>
> If you are not feeding route views or similar community projects,
please
> consider doing so. It helps paint the view for those doing analysis.
>
> - Jared
>
>
> ------------------------------
>
> Message: 5
> Date: Mon, 7 Nov 2011 16:33:15 +0100
> From: Pierre-Yves Maunier <nanog at maunier.org>
> To: Tom Hill <tom at ninjabadger.net>
> Cc: nanog at nanog.org
> Subject: Re: TATA problems?
> Message-ID:
>
<CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> 2011/11/7 Tom Hill <tom at ninjabadger.net>
>
> > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > We seem to be having some problems with our tata links - first
seen in
> EU
> > > about 45 minutes ago, now we're seeing problems in NA. I'm
focused on
> > DNS,
> > > so I'm seeing a lot of timeouts/servfails, but our networking
folks
> are
> > > talking about links dropping.
> > >
> > > Anyone else seeing oddness on the NA Internet right now?
> > >
> > > http://downrightnow.com/ confirms - something is up.
> >
> > There are widespread issues across the Internet; certain versions of
> > Juniper firmware have core dumped after seeing a particular BGP
'UPDATE'
> > message.
> >
> > (That's the running theory at least).
> >
> > It's affected multiple service providers, globally, not just those
> > connected to TATA.
> >
> > Tom
> >
> >
> >
> On our side all our 10.3R2.11 core dumped which made all our
interfaces
> flapped.
> I've been told 10.4R1.9 is affected too.
>
> --
> Pierre-Yves Maunier
>
>
> ------------------------------
>
> Message: 6
> Date: Mon, 7 Nov 2011 15:45:18 +0000
> From: Leigh Porter <leigh.porter at ukbroadband.com>
> To: Pierre-Yves Maunier <nanog at maunier.org>
> Cc: "nanog at nanog.org" <nanog at nanog.org>
> Subject: Re: TATA problems?
> Message-ID: <7994AF08-0622-434F-974F-FC9269469176 at ukbroadband.com>
> Content-Type: text/plain; charset="us-ascii"
>
>
> My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd
> seemed stable.
>
> --
> Leigh
>
>
> On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog at maunier.org>
wrote:
>
> > 2011/11/7 Tom Hill <tom at ninjabadger.net>
> >
> >> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> >>> We seem to be having some problems with our tata links - first
seen in
> EU
> >>> about 45 minutes ago, now we're seeing problems in NA. I'm
focused on
> >> DNS,
> >>> so I'm seeing a lot of timeouts/servfails, but our networking
folks
> are
> >>> talking about links dropping.
> >>>
> >>> Anyone else seeing oddness on the NA Internet right now?
> >>>
> >>> http://downrightnow.com/ confirms - something is up.
> >>
> >> There are widespread issues across the Internet; certain versions
of
> >> Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
> >> message.
> >>
> >> (That's the running theory at least).
> >>
> >> It's affected multiple service providers, globally, not just those
> >> connected to TATA.
> >>
> >> Tom
> >>
> >>
> >>
> > On our side all our 10.3R2.11 core dumped which made all our
interfaces
> > flapped.
> > I've been told 10.4R1.9 is affected too.
> >
> > --
> > Pierre-Yves Maunier
> >
> >
______________________________________________________________________
> > This email has been scanned by the MessageLabs Email Security
System.
> > For more information please visit http://www.messagelabs.com/email
> >
______________________________________________________________________
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
>
>
> ------------------------------
>
> Message: 7
> Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST)
> From: Joe Greco <jgreco at ns.sol.net>
> To: ppauly at gmail.com (Peter Pauly)
> Cc: nanog at nanog.org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <201111071554.pA7FsPHb045359 at aurora.sol.net>
> Content-Type: text/plain; charset=us-ascii
>
> > Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> > from it too and calls to the NOC have not been answered so far...
does
> > anyone have any further information?
> >
> > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>
> Actually, it looks to me like they mean "Time Warner", because that's
> what they said.
>
> The company once known as "Time Warner Telecom" has always been a
> different entity, and hasn't been known as that in some time, now
> being called "twtelecom." Much of that company is what was once
> known as inc.net, a Milwaukee area provider of the '90's.
>
> Time Warner Cable appears to have experienced an implosion this
morning,
> being out of service for about 11 minutes. During that time, packets
> originating here in Milwaukee quickly died in Chicago;
>
> 1 76.46.192.1 8.320 ms 9.900 ms 7.974 ms
> 2 24.160.230.32 7.967 ms 5.975 ms 8.479 ms
> 3 24.160.229.132 8.471 ms 7.969 ms 10.991 ms
> 4 24.160.229.193 9.972 ms 9.973 ms
> 24.160.229.197 9.985 ms
> 5 * * *
> 6 * * *
>
> while packets destined for RR all seemed to be headed out to SJC, from
> what I can tell.
>
> ... JG
> --
> Joe Greco - sol.net Network Services - Milwaukee, WI -
http://www.sol.net
> "We call it the 'one bite at the apple' rule. Give me one chance [and]
> then I
> won't contact you again." - Direct Marketing Ass'n position on e-mail
> spam(CNN)
> With 24 million small businesses in the US alone, that's way too many
> apples.
>
>
>
> ------------------------------
>
> Message: 8
> Date: Mon, 7 Nov 2011 07:55:33 -0800
> From: Kelly Kane <kelly at hawknetworks.com>
> To: Tim Vollebregt <tim at interworx.nl>
> Cc: nanog at nanog.org
> Subject: Re: TATA problems?
> Message-ID:
>
<CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim at interworx.nl> wrote:
> >
> > On #IX there are rumours about Junos version 10.3R2.11 being core
dumped
> and
> > rebooted, which makes sense.
>
> Perhaps related to Juniper PSN-2011-08-327? Did the whole router
> reboot, or just the service module?
>
> We saw one TATA session, and one Abovenet session flap.
>
> Kelly
>
>
>
> ------------------------------
>
> Message: 9
> Date: Mon, 07 Nov 2011 10:02:13 -0600
> From: Blake Hudson <blake at ispn.net>
> To: nanog at nanog.org
> Subject: Re: Time Warner Telecom problems
> Message-ID: <4EB80105.8060703 at ispn.net>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
>
> Joe Greco wrote the following on 11/7/2011 9:54 AM:
> >> Gizmodo is reporting problems at Time Warner Telecom .... we're
> suffering
> >> from it too and calls to the NOC have not been answered so far...
does
> >> anyone have any further information?
> >>
> >> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > Actually, it looks to me like they mean "Time Warner", because
that's
> > what they said.
> >
> > The company once known as "Time Warner Telecom" has always been a
> > different entity, and hasn't been known as that in some time, now
> > being called "twtelecom." Much of that company is what was once
> > known as inc.net, a Milwaukee area provider of the '90's.
> >
> > Time Warner Cable appears to have experienced an implosion this
morning,
> > being out of service for about 11 minutes. During that time,
packets
> > originating here in Milwaukee quickly died in Chicago;
>
> Using the looking glass from TWtelecom, we saw 30-60min outage
(roughly
> 8:30AM to 9:30AM CST) between the Kansas City location and our own
> server room in Kansas City. Other TWtelecom locations appeared to be
> unaffected. Perhaps TWtelecom is served by Timewarner or shares
> equipment in KC. Either way, none of our KC customers who were served
> via TWtelecom or Timewarner were able to reach us. Packets would hit
> Level 3 Communications and die in either direction at the border
between
> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> http://lglass.twtelecom.net/
>
>
>
> ------------------------------
>
> Message: 10
> Date: Mon, 7 Nov 2011 11:04:59 -0500
> From: "Thomas York" <straterra at fuhell.com>
> To: "'Blake Hudson'" <blake at ispn.net>, <nanog at nanog.org>
> Subject: RE: Time Warner Telecom problems
> Message-ID: <[email protected]>
> Content-Type: text/plain; charset="utf-8"
>
> FWIW, We saw issues here in Indianapolis between TWTC and L3 up until
a
> few minutes ago.
>
> --Thomas York
>
> -----Original Message-----
> From: Blake Hudson [mailto:blake at ispn.net]
> Sent: Monday, November 07, 2011 11:02 AM
> To: nanog at nanog.org
> Subject: Re: Time Warner Telecom problems
>
>
> Joe Greco wrote the following on 11/7/2011 9:54 AM:
> >> Gizmodo is reporting problems at Time Warner Telecom .... we're
> >> suffering from it too and calls to the NOC have not been answered
so
> >> far... does anyone have any further information?
> >>
> >> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > Actually, it looks to me like they mean "Time Warner", because
that's
> > what they said.
> >
> > The company once known as "Time Warner Telecom" has always been a
> > different entity, and hasn't been known as that in some time, now
> > being called "twtelecom." Much of that company is what was once
known
> > as inc.net, a Milwaukee area provider of the '90's.
> >
> > Time Warner Cable appears to have experienced an implosion this
> > morning, being out of service for about 11 minutes. During that
time,
> > packets originating here in Milwaukee quickly died in Chicago;
>
> Using the looking glass from TWtelecom, we saw 30-60min outage
(roughly
> 8:30AM to 9:30AM CST) between the Kansas City location and our own
server
> room in Kansas City. Other TWtelecom locations appeared to be
unaffected.
> Perhaps TWtelecom is served by Timewarner or shares equipment in KC.
> Either way, none of our KC customers who were served via TWtelecom or
> Timewarner were able to reach us. Packets would hit Level 3
Communications
> and die in either direction at the border between
> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> http://lglass.twtelecom.net/
>
>
>
>
>
> End of NANOG Digest, Vol 46, Issue 15
> *************************************
More information about the NANOG
mailing list