NANOG Digest, Vol 46, Issue 15
Suess13
suess13 at cfl.rr.com
Tue Nov 8 15:42:58 UTC 2011
Juniper core dump issue, patch is on the way.
On Nov 7, 2011, at 11:41 AM, "Steve Dispensa" <dispensa at phonefactor.com> wrote:
> Level 3 was down in KC, Chi, and San Jose (at least) for us between
> about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.
>
> -Steve
>
>> -----Original Message-----
>> From: nanog-request at nanog.org [mailto:nanog-request at nanog.org]
>> Sent: Monday, November 07, 2011 10:05 AM
>> To: nanog at nanog.org
>> Subject: NANOG Digest, Vol 46, Issue 15
>>
>> Send NANOG mailing list submissions to
>> nanog at nanog.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://mailman.nanog.org/mailman/listinfo/nanog
>> or, via email, send a message with subject or body 'help' to
>> nanog-request at nanog.org
>>
>> You can reach the person managing the list at
>> nanog-owner at nanog.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of NANOG digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Time Warner Telecom problems (Jon Lewis)
>> 2. Re: Performance Issues - PTR Records (Leigh Porter)
>> 3. Re: Time Warner Telecom problems (Ray Van Dolson)
>> 4. General Internet Instability (Jared Mauch)
>> 5. Re: TATA problems? (Pierre-Yves Maunier)
>> 6. Re: TATA problems? (Leigh Porter)
>> 7. Re: Time Warner Telecom problems (Joe Greco)
>> 8. Re: TATA problems? (Kelly Kane)
>> 9. Re: Time Warner Telecom problems (Blake Hudson)
>> 10. RE: Time Warner Telecom problems (Thomas York)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST)
>> From: Jon Lewis <jlewis at lewis.org>
>> To: Peter Pauly <ppauly at gmail.com>
>> Cc: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <Pine.LNX.4.61.1111071005540.24418 at soloth.lewis.org>
>> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
>>
>> On Mon, 7 Nov 2011, Peter Pauly wrote:
>>
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>>
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>
>> I noticed just a little while ago that we're having a lot of DNS fail.
>> Initial findings were that several of the root-servers we were trying
> to
>> reach via our TWTelecom link were unreachable after 2 hops into TWT.
>>
>> 4 64-128-130-233.static.twtelecom.NET (64.128.130.233) 2.399 ms
> 2.298
>> ms 2.338 ms
>> 5 mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18) 11.571 ms
>> 11.552 ms 9.467 ms
>> 6 * * *
>> 7 * * *
>> 8 * * *
>>
>> For instance, a.root-servers.net is pingable from a rackspace server,
> but
>> not from our network (unless I shut off TWT, at which point it is, but
>> it's apparently not the same a.root-servers.net instance rackspace
> sees).
>> I assume this is one of the root-servers being anycast.
>>
>> Shutting off our BGP with TWT didn't appear to help (though the
>> root-servers became reachable)...so I assume there's more going on
> than
>> just TWT routing fail.
>>
>> ----------------------------------------------------------------------
>> Jon Lewis, MCP :) | I route
>> Senior Network Engineer | therefore you are
>> Atlantic Net |
>> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Mon, 7 Nov 2011 15:29:30 +0000
>> From: Leigh Porter <leigh.porter at ukbroadband.com>
>> To: Bj?rn Mork <bjorn at mork.no>
>> Cc: "nanog at nanog.org" <nanog at nanog.org>
>> Subject: Re: Performance Issues - PTR Records
>> Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863 at ukbroadband.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>>
>> On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn at mork.no> wrote:
>>
>>> Leigh Porter <leigh.porter at ukbroadband.com> writes:
>>>
>>>> Indeed, there is no way I would allow that either. But really,
>>>> providing a reverse zone and forward zone to match is a case of
> five
>>>> minutes and a shell script or a DNS that as Steinar said, will
>>>> synthesise results.
>>>>
>>>> It's really not all that difficult..
>>>
>>> No, not at all. It's just totally pointless. Any IPv6 address is
> just
>>> as pretty as a synthesized name. Maybe even prettier. Do you prefer
>>> "2001:db8:1::2" or
> "20010db8000100000000000000000002.rev.example.com"?
>>>
>>> If we're going to provide any reverse DNS for end users, then it is
>>> because we can create names which actually improves something.
>>>
>>>
>>> Bj?rn
>>>
>>>
>>
>> Yup it is pointless.. Mine are all ipadrress.domain which is of
> course,
>> pointless.. I suppose at least somebody would glean that perhaps its a
>> home user rather than a business or server on that address but that's
> all.
>>
>> With IPv6 arguably even more pointless as you say.
>>
>> ______________________________________________________________________
>> This email has been scanned by the MessageLabs Email Security System.
>> For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>>
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Mon, 7 Nov 2011 07:28:18 -0800
>> From: Ray Van Dolson <rvandolson at esri.com>
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <20111107152817.GA29715 at esri.com>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>>
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>
>> FWIW, my home TWC connection dropped this morning for about 15 minutes
>> (Southern California around 6:30AM'ish). Still could ping the default
>> gateway, but packets weren't traversing much beyond that.
>>
>> Didn't investigate further, just headed into work.
>>
>> Ray
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Mon, 7 Nov 2011 10:31:31 -0500
>> From: Jared Mauch <jared at puck.nether.net>
>> To: Tom Hill <tom at ninjabadger.net>
>> Cc: nanog at nanog.org
>> Subject: General Internet Instability
>> Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD at puck.nether.net>
>> Content-Type: text/plain; charset=us-ascii
>>
>> On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
>>
>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>> We seem to be having some problems with our tata links - first seen
> in
>> EU
>>>> about 45 minutes ago, now we're seeing problems in NA. I'm focused
> on
>> DNS,
>>>> so I'm seeing a lot of timeouts/servfails, but our networking folks
> are
>>>> talking about links dropping.
>>>>
>>>> Anyone else seeing oddness on the NA Internet right now?
>>>>
>>>> http://downrightnow.com/ confirms - something is up.
>>>
>>> There are widespread issues across the Internet; certain versions of
>>> Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
>>> message.
>>>
>>> (That's the running theory at least).
>>>
>>> It's affected multiple service providers, globally, not just those
>>> connected to TATA.
>>
>>
>> Pretty much any major BGP event will impact multiple providers.
>>
>> A threshold you should use to view the general instability (which I
> find
>> valuable, you may as well) is route views data.
>>
>> If you look at the BGP UPDATES archive sizes, you can see when
> something
>> happens, e.g.:
>>
>> http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
>>
>> Take a look at the size of the updates.20111107.1400.bz2 file and the
> 1415
>> file. They are abnormally large compared to a normal period of time.
>> This shows there were a lot of updates out there being processed and a
>> reference to levels of instability.
>>
>> If you are not feeding route views or similar community projects,
> please
>> consider doing so. It helps paint the view for those doing analysis.
>>
>> - Jared
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Mon, 7 Nov 2011 16:33:15 +0100
>> From: Pierre-Yves Maunier <nanog at maunier.org>
>> To: Tom Hill <tom at ninjabadger.net>
>> Cc: nanog at nanog.org
>> Subject: Re: TATA problems?
>> Message-ID:
>>
> <CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA at mail.gmail.com>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> 2011/11/7 Tom Hill <tom at ninjabadger.net>
>>
>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>> We seem to be having some problems with our tata links - first
> seen in
>> EU
>>>> about 45 minutes ago, now we're seeing problems in NA. I'm
> focused on
>>> DNS,
>>>> so I'm seeing a lot of timeouts/servfails, but our networking
> folks
>> are
>>>> talking about links dropping.
>>>>
>>>> Anyone else seeing oddness on the NA Internet right now?
>>>>
>>>> http://downrightnow.com/ confirms - something is up.
>>>
>>> There are widespread issues across the Internet; certain versions of
>>> Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
>>> message.
>>>
>>> (That's the running theory at least).
>>>
>>> It's affected multiple service providers, globally, not just those
>>> connected to TATA.
>>>
>>> Tom
>>>
>>>
>>>
>> On our side all our 10.3R2.11 core dumped which made all our
> interfaces
>> flapped.
>> I've been told 10.4R1.9 is affected too.
>>
>> --
>> Pierre-Yves Maunier
>>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Mon, 7 Nov 2011 15:45:18 +0000
>> From: Leigh Porter <leigh.porter at ukbroadband.com>
>> To: Pierre-Yves Maunier <nanog at maunier.org>
>> Cc: "nanog at nanog.org" <nanog at nanog.org>
>> Subject: Re: TATA problems?
>> Message-ID: <7994AF08-0622-434F-974F-FC9269469176 at ukbroadband.com>
>> Content-Type: text/plain; charset="us-ascii"
>>
>>
>> My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd
>> seemed stable.
>>
>> --
>> Leigh
>>
>>
>> On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog at maunier.org>
> wrote:
>>
>>> 2011/11/7 Tom Hill <tom at ninjabadger.net>
>>>
>>>> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>>>>> We seem to be having some problems with our tata links - first
> seen in
>> EU
>>>>> about 45 minutes ago, now we're seeing problems in NA. I'm
> focused on
>>>> DNS,
>>>>> so I'm seeing a lot of timeouts/servfails, but our networking
> folks
>> are
>>>>> talking about links dropping.
>>>>>
>>>>> Anyone else seeing oddness on the NA Internet right now?
>>>>>
>>>>> http://downrightnow.com/ confirms - something is up.
>>>>
>>>> There are widespread issues across the Internet; certain versions
> of
>>>> Juniper firmware have core dumped after seeing a particular BGP
>> 'UPDATE'
>>>> message.
>>>>
>>>> (That's the running theory at least).
>>>>
>>>> It's affected multiple service providers, globally, not just those
>>>> connected to TATA.
>>>>
>>>> Tom
>>>>
>>>>
>>>>
>>> On our side all our 10.3R2.11 core dumped which made all our
> interfaces
>>> flapped.
>>> I've been told 10.4R1.9 is affected too.
>>>
>>> --
>>> Pierre-Yves Maunier
>>>
>>>
> ______________________________________________________________________
>>> This email has been scanned by the MessageLabs Email Security
> System.
>>> For more information please visit http://www.messagelabs.com/email
>>>
> ______________________________________________________________________
>>
>> ______________________________________________________________________
>> This email has been scanned by the MessageLabs Email Security System.
>> For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>>
>>
>>
>> ------------------------------
>>
>> Message: 7
>> Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST)
>> From: Joe Greco <jgreco at ns.sol.net>
>> To: ppauly at gmail.com (Peter Pauly)
>> Cc: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <201111071554.pA7FsPHb045359 at aurora.sol.net>
>> Content-Type: text/plain; charset=us-ascii
>>
>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>> from it too and calls to the NOC have not been answered so far...
> does
>>> anyone have any further information?
>>>
>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>
>> Actually, it looks to me like they mean "Time Warner", because that's
>> what they said.
>>
>> The company once known as "Time Warner Telecom" has always been a
>> different entity, and hasn't been known as that in some time, now
>> being called "twtelecom." Much of that company is what was once
>> known as inc.net, a Milwaukee area provider of the '90's.
>>
>> Time Warner Cable appears to have experienced an implosion this
> morning,
>> being out of service for about 11 minutes. During that time, packets
>> originating here in Milwaukee quickly died in Chicago;
>>
>> 1 76.46.192.1 8.320 ms 9.900 ms 7.974 ms
>> 2 24.160.230.32 7.967 ms 5.975 ms 8.479 ms
>> 3 24.160.229.132 8.471 ms 7.969 ms 10.991 ms
>> 4 24.160.229.193 9.972 ms 9.973 ms
>> 24.160.229.197 9.985 ms
>> 5 * * *
>> 6 * * *
>>
>> while packets destined for RR all seemed to be headed out to SJC, from
>> what I can tell.
>>
>> ... JG
>> --
>> Joe Greco - sol.net Network Services - Milwaukee, WI -
> http://www.sol.net
>> "We call it the 'one bite at the apple' rule. Give me one chance [and]
>> then I
>> won't contact you again." - Direct Marketing Ass'n position on e-mail
>> spam(CNN)
>> With 24 million small businesses in the US alone, that's way too many
>> apples.
>>
>>
>>
>> ------------------------------
>>
>> Message: 8
>> Date: Mon, 7 Nov 2011 07:55:33 -0800
>> From: Kelly Kane <kelly at hawknetworks.com>
>> To: Tim Vollebregt <tim at interworx.nl>
>> Cc: nanog at nanog.org
>> Subject: Re: TATA problems?
>> Message-ID:
>>
> <CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>>
>> On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim at interworx.nl> wrote:
>>>
>>> On #IX there are rumours about Junos version 10.3R2.11 being core
> dumped
>> and
>>> rebooted, which makes sense.
>>
>> Perhaps related to Juniper PSN-2011-08-327? Did the whole router
>> reboot, or just the service module?
>>
>> We saw one TATA session, and one Abovenet session flap.
>>
>> Kelly
>>
>>
>>
>> ------------------------------
>>
>> Message: 9
>> Date: Mon, 07 Nov 2011 10:02:13 -0600
>> From: Blake Hudson <blake at ispn.net>
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>> Message-ID: <4EB80105.8060703 at ispn.net>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>>
>>
>> Joe Greco wrote the following on 11/7/2011 9:54 AM:
>>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>> suffering
>>>> from it too and calls to the NOC have not been answered so far...
> does
>>>> anyone have any further information?
>>>>
>>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>> Actually, it looks to me like they mean "Time Warner", because
> that's
>>> what they said.
>>>
>>> The company once known as "Time Warner Telecom" has always been a
>>> different entity, and hasn't been known as that in some time, now
>>> being called "twtelecom." Much of that company is what was once
>>> known as inc.net, a Milwaukee area provider of the '90's.
>>>
>>> Time Warner Cable appears to have experienced an implosion this
> morning,
>>> being out of service for about 11 minutes. During that time,
> packets
>>> originating here in Milwaukee quickly died in Chicago;
>>
>> Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
>> 8:30AM to 9:30AM CST) between the Kansas City location and our own
>> server room in Kansas City. Other TWtelecom locations appeared to be
>> unaffected. Perhaps TWtelecom is served by Timewarner or shares
>> equipment in KC. Either way, none of our KC customers who were served
>> via TWtelecom or Timewarner were able to reach us. Packets would hit
>> Level 3 Communications and die in either direction at the border
> between
>> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
>> http://lglass.twtelecom.net/
>>
>>
>>
>> ------------------------------
>>
>> Message: 10
>> Date: Mon, 7 Nov 2011 11:04:59 -0500
>> From: "Thomas York" <straterra at fuhell.com>
>> To: "'Blake Hudson'" <blake at ispn.net>, <nanog at nanog.org>
>> Subject: RE: Time Warner Telecom problems
>> Message-ID: <[email protected]>
>> Content-Type: text/plain; charset="utf-8"
>>
>> FWIW, We saw issues here in Indianapolis between TWTC and L3 up until
> a
>> few minutes ago.
>>
>> --Thomas York
>>
>> -----Original Message-----
>> From: Blake Hudson [mailto:blake at ispn.net]
>> Sent: Monday, November 07, 2011 11:02 AM
>> To: nanog at nanog.org
>> Subject: Re: Time Warner Telecom problems
>>
>>
>> Joe Greco wrote the following on 11/7/2011 9:54 AM:
>>>> Gizmodo is reporting problems at Time Warner Telecom .... we're
>>>> suffering from it too and calls to the NOC have not been answered
> so
>>>> far... does anyone have any further information?
>>>>
>>>> http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
>>> Actually, it looks to me like they mean "Time Warner", because
> that's
>>> what they said.
>>>
>>> The company once known as "Time Warner Telecom" has always been a
>>> different entity, and hasn't been known as that in some time, now
>>> being called "twtelecom." Much of that company is what was once
> known
>>> as inc.net, a Milwaukee area provider of the '90's.
>>>
>>> Time Warner Cable appears to have experienced an implosion this
>>> morning, being out of service for about 11 minutes. During that
> time,
>>> packets originating here in Milwaukee quickly died in Chicago;
>>
>> Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
>> 8:30AM to 9:30AM CST) between the Kansas City location and our own
> server
>> room in Kansas City. Other TWtelecom locations appeared to be
> unaffected.
>> Perhaps TWtelecom is served by Timewarner or shares equipment in KC.
>> Either way, none of our KC customers who were served via TWtelecom or
>> Timewarner were able to reach us. Packets would hit Level 3
> Communications
>> and die in either direction at the border between
>> L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
>> http://lglass.twtelecom.net/
>>
>>
>>
>>
>>
>> End of NANOG Digest, Vol 46, Issue 15
>> *************************************
>
More information about the NANOG
mailing list