Level3 worldwide emergency upgrade?

Siegel, David David.Siegel at Level3.com
Wed Feb 6 17:01:22 UTC 2013


Hi Ray,

This topic reminds me of yesterday's discussion in the conference around getting some BCOP's drafted.  it would be useful to confirm my own view of the BCOP around communicating security issues.  My understanding for the best practice is to limit knowledge distribution of security related problems both before and after the patches are deployed.  You limit knowledge before the patch is deployed to prevent yourself from being exploited, but you also limit knowledge afterwards in order to limit potential damage to others (customers, competitors...the Internet at large).  You also do not want to announce that you will be deploying a security patch until you have a fix in hand and know when you will deploy it (typically, next available maintenance window unless the cat is out of the bag and danger is real and imminent).

As a service provider, you should stay on top of security alerts from your vendors so that you can make your own decision about what action is required.  I would not recommend relying on service provider maintenance bulletins or public operations mailing lists for obtaining this type of information.  There is some information that can cause more harm than good if it is distributed in the wrong way and information relating to security vulnerabilities definitely falls into that category.

Dave

-----Original Message-----
From: Ray Wong [mailto:rayw at rayw.net] 
Sent: Wednesday, February 06, 2013 9:16 AM
To: nanog at nanog.org
Subject: Re: Level3 worldwide emergency upgrade?

>

OK, having had that first cup of coffee, I can say perhaps the main reason I was wondering is I've gotten used to Level3 always being on top of things (and admittedly, rarely communicating). They've reached the top by often being a black box of reliability, so it's (perhaps
unrealistically) surprising to see them caught by surprise. Anything that pushes them into scramble mode causes me to lose a little sleep anyway. The alternative to what they did seems likely for at least a few providers who'll NOT manage to fix things in time, so I may well be looking at longer outages from other providers, and need to issue guidance to others on what to do if/when other links go down for periods long enough that all the cost-bounding monitoring alarms start to scream even louder.

I was also grumpy at myself for having not noticed advance communication, which I still don't seem to have, though since I outsourced my email to bigG, I've noticed I'm more likely to miss things. Perhaps giving up maintaining that massive set of procmail rules has cost me a bit more edge.

Related, of course, just because you design/run your network to tolerate some issues doesn't mean you can also budget to be in support contract as well. :) Knowing more about the exploit/fix might mean trying to find a way to get free upgrades to some kit to prevent more localized attacks to other types of gear, as well, though in this case it's all about Juniper PR839412 then, so vendor specific, it seems?

There are probably more reasons to wish for more info, too. There's still more of them (exploiters/attackers) than there are those of us trying to keep things running smoothly and transparently, so anything that smells of "OMG new exploit found!" also triggers my desire to share information. The network bad guys share information far more quickly and effectively than we do, it often seems.

-R>





More information about the NANOG mailing list