NANOG Digest, Vol 46, Issue 15

Suess13 suess13 at cfl.rr.com
Tue Nov 8 09:42:58 CST 2011


Juniper core dump issue, patch is on the way.



On Nov 7, 2011, at 11:41 AM, "Steve Dispensa" <dispensa at phonefactor.com> wrote:

> Level 3 was down in KC, Chi, and San Jose (at least) for us between
> about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.
> 
> -Steve
> 
>> -----Original Message-----
>> From: nanog-request at nanog.org [mailto:nanog-request at nanog.org]
>> Sent: Monday, November 07, 2011 10:05 AM
>> To: nanog at nanog.org
>> Subject: NANOG Digest, Vol 46, Issue 15
>> 
>> Send NANOG mailing list submissions to
>>    nanog at nanog.org
>> 
>> To subscribe or unsubscribe via the World Wide Web, visit
>>    https://mailman.nanog.org/mailman/listinfo/nanog
>> or, via email, send a message with subject or body 'help' to
>>    nanog-request at nanog.org
>> 
>> You can reach the person managing the list at
>>    nanog-owner at nanog.org
>> 
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of NANOG digest..."
>> 
>> 
>> Today's Topics:
>> 
>>   1. Re: Time Warner Telecom problems (Jon Lewis)
>>   2. Re: Performance Issues - PTR Records (Leigh Porter)
>>   3. Re: Time Warner Telecom problems (Ray Van Dolson)
>>   4. General Internet Instability (Jared Mauch)
>>   5. Re: TATA problems? (Pierre-Yves Maunier)
>>   6. Re: TATA problems? (Leigh Porter)
>>   7. Re: Time Warner Telecom problems (Joe Greco)
>>   8. Re: TATA problems? (Kelly Kane)
>>   9. Re: Time Warner Telecom problems (Blake Hudson)
>>  10. RE: Time Warner Telecom problems (Thomas York)
>> 
>> 
>> ----------------------------------------------------------------------
>> 
>> Message: 1
>> Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST)
>> From: Jon Lewis <jlewis at lewis.org>
>> To: Peter Pauly <ppauly at gmail.com>
>> Cc: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <Pine.LNX.4.61.1111071005540.24418 at soloth.lewis.org>
>> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>> 
>> On Mon, 7 Nov 2011, Peter Pauly wrote:
>> 
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>> 
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>> 
>> I noticed just a little while ago that we're having a lot of DNS fail.
>> Initial findings were that several of the root-servers we were trying
> to
>> reach via our TWTelecom link were unreachable after 2 hops into TWT.
>> 
>>  4  64-128-130-233.static.twtelecom.NET (64.128.130.233)  2.399 ms
> 2.298
>> ms  2.338 ms
>>  5  mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18)  11.571 ms
>> 11.552 ms  9.467 ms
>>  6  * * *
>>  7  * * *
>>  8  * * *
>> 
>> For instance, a.root-servers.net is pingable from a rackspace server,
> but
>> not from our network (unless I shut off TWT, at which point it is, but
>> it's apparently not the same a.root-servers.net instance rackspace
> sees).
>> I assume this is one of the root-servers being anycast.
>> 
>> Shutting off our BGP with TWT didn't appear to help (though the
>> root-servers became reachable)...so I assume there's more going on
> than
>> just TWT routing fail.
>> 
>> ----------------------------------------------------------------------
>>  Jon Lewis, MCP :)           |  I route
>>  Senior Network Engineer     |  therefore you are
>>  Atlantic Net                |
>> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 2
>> Date: Mon, 7 Nov 2011 15:29:30 +0000
>> From: Leigh Porter <leigh.porter at ukbroadband.com>
>> To: Bj?rn Mork <bjorn at mork.no>
>> Cc: "nanog at nanog.org" <nanog at nanog.org>
>> Subject: Re: Performance Issues - PTR Records
>> Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863 at ukbroadband.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>> 
>> 
>> 
>> On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn at mork.no> wrote:
>> 
>>> Leigh Porter <leigh.porter at ukbroadband.com> writes:
>>> 
>>>> Indeed, there is no way I would allow that either. But really,
>>>> providing a reverse zone and forward zone to match is a case of
> five
>>>> minutes and a shell script or a DNS that as Steinar said, will
>>>> synthesise results.
>>>> 
>>>> It's really not all that difficult..
>>> 
>>> No, not at all.  It's just totally pointless.  Any IPv6 address is
> just
>>> as pretty as a synthesized name.  Maybe even prettier. Do you prefer
>>> "2001:db8:1::2" or
> "20010db8000100000000000000000002.rev.example.com"?
>>> 
>>> If we're going to provide any reverse DNS for end users, then it is
>>> because we can create names which actually improves something.
>>> 
>>> 
>>> Bj?rn
>>> 
>>> 
>> 
>> Yup it is pointless.. Mine are all ipadrress.domain which is of
> course,
>> pointless.. I suppose at least somebody would glean that perhaps its a
>> home user rather than a business or server on that address but that's
> all.
>> 
>> With IPv6 arguably even more pointless as you say.
>> 
>> ______________________________________________________________________
>> This email has been scanned by the MessageLabs Email Security System.
>> For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 3
>> Date: Mon, 7 Nov 2011 07:28:18 -0800
>> From: Ray Van Dolson <rvandolson at esri.com>
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <20111107152817.GA29715 at esri.com>
>> Content-Type: text/plain; charset=us-ascii
>> 
>> On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>> 
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>> 
>> FWIW, my home TWC connection dropped this morning for about 15 minutes
>> (Southern California around 6:30AM'ish).  Still could ping the default
>> gateway, but packets weren't traversing much beyond that.
>> 
>> Didn't investigate further, just headed into work.
>> 
>> Ray
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 4
>> Date: Mon, 7 Nov 2011 10:31:31 -0500
>> From: Jared Mauch <jared at puck.nether.net>
>> To: Tom Hill <tom at ninjabadger.net>
>> Cc: nanog at nanog.org
>> Subject: General Internet Instability
>> Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD at puck.nether.net>
>> Content-Type: text/plain; charset=us-ascii
>> 
>> On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
>> 
>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>> We seem to be having some problems with our tata links - first seen
> in
>> EU
>>>> about 45 minutes ago, now we're seeing problems in NA.  I'm focused
> on
>> DNS,
>>>> so I'm seeing a lot of timeouts/servfails, but our networking folks
> are
>>>> talking about links dropping.
>>>> 
>>>> Anyone else seeing oddness on the NA Internet right now?
>>>> 
>>>> http://downrightnow.com/ confirms - something is up.
>>> 
>>> There are widespread issues across the Internet; certain versions of
>>> Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
>>> message.
>>> 
>>> (That's the running theory at least).
>>> 
>>> It's affected multiple service providers, globally, not just those
>>> connected to TATA.
>> 
>> 
>> Pretty much any major BGP event will impact multiple providers.
>> 
>> A threshold you should use to view the general instability (which I
> find
>> valuable, you may as well) is route views data.
>> 
>> If you look at the BGP UPDATES archive sizes, you can see when
> something
>> happens, e.g.:
>> 
>> http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
>> 
>> Take a look at the size of the updates.20111107.1400.bz2 file and the
> 1415
>> file.  They are abnormally large compared to a normal period of time.
>> This shows there were a lot of updates out there being processed and a
>> reference to levels of instability.
>> 
>> If you are not feeding route views or similar community projects,
> please
>> consider doing so.  It helps paint the view for those doing analysis.
>> 
>> - Jared
>> 
>> 
>> ------------------------------
>> 
>> Message: 5
>> Date: Mon, 7 Nov 2011 16:33:15 +0100
>> From: Pierre-Yves Maunier <nanog at maunier.org>
>> To: Tom Hill <tom at ninjabadger.net>
>> Cc: nanog at nanog.org
>> Subject: Re: TATA problems?
>> Message-ID:
>> 
> <CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>> 
>> 2011/11/7 Tom Hill <tom at ninjabadger.net>
>> 
>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>> We seem to be having some problems with our tata links - first
> seen in
>> EU
>>>> about 45 minutes ago, now we're seeing problems in NA.  I'm
> focused on
>>> DNS,
>>>> so I'm seeing a lot of timeouts/servfails, but our networking
> folks
>> are
>>>> talking about links dropping.
>>>> 
>>>> Anyone else seeing oddness on the NA Internet right now?
>>>> 
>>>> http://downrightnow.com/ confirms - something is up.
>>> 
>>> There are widespread issues across the Internet; certain versions of
>>> Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
>>> message.
>>> 
>>> (That's the running theory at least).
>>> 
>>> It's affected multiple service providers, globally, not just those
>>> connected to TATA.
>>> 
>>> Tom
>>> 
>>> 
>>> 
>> On our side all our 10.3R2.11 core dumped which made all our
> interfaces
>> flapped.
>> I've been told 10.4R1.9 is affected too.
>> 
>> --
>> Pierre-Yves Maunier
>> 
>> 
>> ------------------------------
>> 
>> Message: 6
>> Date: Mon, 7 Nov 2011 15:45:18 +0000
>> From: Leigh Porter <leigh.porter at ukbroadband.com>
>> To: Pierre-Yves Maunier <nanog at maunier.org>
>> Cc: "nanog at nanog.org" <nanog at nanog.org>
>> Subject: Re: TATA problems?
>> Message-ID: <7994AF08-0622-434F-974F-FC9269469176 at ukbroadband.com>
>> Content-Type: text/plain; charset="us-ascii"
>> 
>> 
>> My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd
>> seemed stable.
>> 
>> --
>> Leigh
>> 
>> 
>> On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog at maunier.org>
> wrote:
>> 
>>> 2011/11/7 Tom Hill <tom at ninjabadger.net>
>>> 
>>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>>> We seem to be having some problems with our tata links - first
> seen in
>> EU
>>>>> about 45 minutes ago, now we're seeing problems in NA.  I'm
> focused on
>>>> DNS,
>>>>> so I'm seeing a lot of timeouts/servfails, but our networking
> folks
>> are
>>>>> talking about links dropping.
>>>>> 
>>>>> Anyone else seeing oddness on the NA Internet right now?
>>>>> 
>>>>> http://downrightnow.com/ confirms - something is up.
>>>> 
>>>> There are widespread issues across the Internet; certain versions
> of
>>>> Juniper firmware have core dumped after seeing a particular BGP
>> 'UPDATE'
>>>> message.
>>>> 
>>>> (That's the running theory at least).
>>>> 
>>>> It's affected multiple service providers, globally, not just those
>>>> connected to TATA.
>>>> 
>>>> Tom
>>>> 
>>>> 
>>>> 
>>> On our side all our 10.3R2.11 core dumped which made all our
> interfaces
>>> flapped.
>>> I've been told 10.4R1.9 is affected too.
>>> 
>>> --
>>> Pierre-Yves Maunier
>>> 
>>> 
> ______________________________________________________________________
>>> This email has been scanned by the MessageLabs Email Security
> System.
>>> For more information please visit http://www.messagelabs.com/email
>>> 
> ______________________________________________________________________
>> 
>> ______________________________________________________________________
>> This email has been scanned by the MessageLabs Email Security System.
>> For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 7
>> Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST)
>> From: Joe Greco <jgreco at ns.sol.net>
>> To: ppauly at gmail.com (Peter Pauly)
>> Cc: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <201111071554.pA7FsPHb045359 at aurora.sol.net>
>> Content-Type: text/plain; charset=us-ascii
>> 
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>> 
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>> 
>> Actually, it looks to me like they mean "Time Warner", because that's
>> what they said.
>> 
>> The company once known as "Time Warner Telecom" has always been a
>> different entity, and hasn't been known as that in some time, now
>> being called "twtelecom."  Much of that company is what was once
>> known as inc.net, a Milwaukee area provider of the '90's.
>> 
>> Time Warner Cable appears to have experienced an implosion this
> morning,
>> being out of service for about 11 minutes.  During that time, packets
>> originating here in Milwaukee quickly died in Chicago;
>> 
>> 1  76.46.192.1  8.320 ms  9.900 ms  7.974 ms
>> 2  24.160.230.32  7.967 ms  5.975 ms  8.479 ms
>> 3  24.160.229.132  8.471 ms  7.969 ms  10.991 ms
>> 4  24.160.229.193  9.972 ms  9.973 ms
>>    24.160.229.197  9.985 ms
>> 5  * * *
>> 6  * * *
>> 
>> while packets destined for RR all seemed to be headed out to SJC, from
>> what I can tell.
>> 
>> ... JG
>> --
>> Joe Greco - sol.net Network Services - Milwaukee, WI -
> http://www.sol.net
>> "We call it the 'one bite at the apple' rule. Give me one chance [and]
>> then I
>> won't contact you again." - Direct Marketing Ass'n position on e-mail
>> spam(CNN)
>> With 24 million small businesses in the US alone, that's way too many
>> apples.
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 8
>> Date: Mon, 7 Nov 2011 07:55:33 -0800
>> From: Kelly Kane <kelly at hawknetworks.com>
>> To: Tim Vollebregt <tim at interworx.nl>
>> Cc: nanog at nanog.org
>> Subject: Re: TATA problems?
>> Message-ID:
>> 
> <CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>> 
>> On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim at interworx.nl> wrote:
>>> 
>>> On #IX there are rumours about Junos version 10.3R2.11 being core
> dumped
>> and
>>> rebooted, which makes sense.
>> 
>> Perhaps related to Juniper PSN-2011-08-327? Did the whole router
>> reboot, or just the service module?
>> 
>> We saw one TATA session, and one Abovenet session flap.
>> 
>> Kelly
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 9
>> Date: Mon, 07 Nov 2011 10:02:13 -0600
>> From: Blake Hudson <blake at ispn.net>
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <4EB80105.8060703 at ispn.net>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>> 
>> 
>> Joe Greco wrote the following on 11/7/2011 9:54 AM:
>>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>>> from it too and calls to the NOC have not been answered so far...
> does
>>>> anyone have any further information?
>>>> 
>>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>> Actually, it looks to me like they mean "Time Warner", because
> that's
>>> what they said.
>>> 
>>> The company once known as "Time Warner Telecom" has always been a
>>> different entity, and hasn't been known as that in some time, now
>>> being called "twtelecom."  Much of that company is what was once
>>> known as inc.net, a Milwaukee area provider of the '90's.
>>> 
>>> Time Warner Cable appears to have experienced an implosion this
> morning,
>>> being out of service for about 11 minutes.  During that time,
> packets
>>> originating here in Milwaukee quickly died in Chicago;
>> 
>> Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
>> 8:30AM to 9:30AM CST) between the Kansas City location and our own
>> server room in Kansas City. Other TWtelecom locations appeared to be
>> unaffected. Perhaps TWtelecom is served by Timewarner or shares
>> equipment in KC. Either way, none of our KC customers who were served
>> via TWtelecom or Timewarner were able to reach us. Packets would hit
>> Level 3 Communications and die in either direction at the border
> between
>> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
>> http://lglass.twtelecom.net/
>> 
>> 
>> 
>> ------------------------------
>> 
>> Message: 10
>> Date: Mon, 7 Nov 2011 11:04:59 -0500
>> From: "Thomas York" <straterra at fuhell.com>
>> To: "'Blake Hudson'" <blake at ispn.net>,    <nanog at nanog.org>
>> Subject: RE: Time Warner Telecom problems
>> Message-ID: <c23401cc9d67$046baea0$0d430be0$@fuhell.com>
>> Content-Type: text/plain;    charset="utf-8"
>> 
>> FWIW, We saw issues here in Indianapolis between TWTC and L3 up until
> a
>> few minutes ago.
>> 
>> --Thomas York
>> 
>> -----Original Message-----
>> From: Blake Hudson [mailto:blake at ispn.net]
>> Sent: Monday, November 07, 2011 11:02 AM
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> 
>> 
>> Joe Greco wrote the following on 11/7/2011 9:54 AM:
>>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>>>> suffering from it too and calls to the NOC have not been answered
> so
>>>> far...  does anyone have any further information?
>>>> 
>>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>> Actually, it looks to me like they mean "Time Warner", because
> that's
>>> what they said.
>>> 
>>> The company once known as "Time Warner Telecom" has always been a
>>> different entity, and hasn't been known as that in some time, now
>>> being called "twtelecom."  Much of that company is what was once
> known
>>> as inc.net, a Milwaukee area provider of the '90's.
>>> 
>>> Time Warner Cable appears to have experienced an implosion this
>>> morning, being out of service for about 11 minutes.  During that
> time,
>>> packets originating here in Milwaukee quickly died in Chicago;
>> 
>> Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
>> 8:30AM to 9:30AM CST) between the Kansas City location and our own
> server
>> room in Kansas City. Other TWtelecom locations appeared to be
> unaffected.
>> Perhaps TWtelecom is served by Timewarner or shares equipment in KC.
>> Either way, none of our KC customers who were served via TWtelecom or
>> Timewarner were able to reach us. Packets would hit Level 3
> Communications
>> and die in either direction at the border between
>> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
>> http://lglass.twtelecom.net/
>> 
>> 
>> 
>> 
>> 
>> End of NANOG Digest, Vol 46, Issue 15
>> *************************************
> 



More information about the NANOG mailing list