General Internet Instability

Mon Nov 7 15:31:31 UTC 2011

On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:

> On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
>> We seem to be having some problems with our tata links - first seen in EU
>> about 45 minutes ago, now we're seeing problems in NA.  I'm focused on DNS,
>> so I'm seeing a lot of timeouts/servfails, but our networking folks are
>> talking about links dropping.
>> 
>> Anyone else seeing oddness on the NA Internet right now?
>> 
>> http://downrightnow.com/ confirms - something is up.
> 
> There are widespread issues across the Internet; certain versions of
> Juniper firmware have core dumped after seeing a particular BGP 'UPDATE'
> message. 
> 
> (That's the running theory at least).
> 
> It's affected multiple service providers, globally, not just those
> connected to TATA.

Pretty much any major BGP event will impact multiple providers.

A threshold you should use to view the general instability (which I find valuable, you may as well) is route views data.

If you look at the BGP UPDATES archive sizes, you can see when something happens, e.g.:

http://archive.routeviews.org/bgpdata/2011.11/UPDATES/

Take a look at the size of the updates.20111107.1400.bz2 file and the 1415 file.  They are abnormally large compared to a normal period of time.  This shows there were a lot of updates out there being processed and a reference to levels of instability.

If you are not feeding route views or similar community projects, please consider doing so.  It helps paint the view for those doing analysis.

- Jared