The DDOS problem & security BOF: Am i mistaken?

Wed Oct 15 20:39:58 UTC 2008

Scott,

On Oct 14, 2008, at 9:08 AM, Scott Doty wrote:

> First, the good news:  so far, the NANOG conference has been very  
> valuable and
> content-rich, covering a lot of issues that need to be discussed.   
> For that, I am grateful.

Thank you. We worked hard to make it valuable.
>
>
> But now, the bad news(?):  Maybe it's just me & my paranoia, but do  
> I detect
> an inkling of "murk spam" going on with some presentations?

Not sure what you mean by "murk spam". Thats a term that died years  
ago. And it really related to people claiming that spam was "in  
compliance with federal laws". But I think I can guess your intentions  
from the tone of your email, so let me try and respond.

>
>
> Because there seems to be a fundamental misunderstanding, either on  
> my part,
> or the part of certain vendors: I'm hear to discuss ideas & freely  
> share
> them, and they are here to discuss (it would seem) their products.  
> Sometimes
> both goals coincide, and that is fine...but...
>
> When a vendor at the security BOF starts showing documents that are  
> "company
> confidential", and trying to whip up a climate of fear, that we  
> should all
> deploy their product in front of our recursive name servers, i get  
> this
> funny feeling that I am being "murk spammed".

Well, that's interesting. I see your last NANOG was 9, in February of  
1997. So "Welcome back!". We're glad to have you here in person.  
Things have changed slightly since then. NSP-SEC never existed in  
1997. It really came about in the early 2000's where it was developed  
as a forum for actual operators to share views and thoughts, generally  
in real time, to help the 'net in general survive disruption,  
malicious or otherwise. It has really worked pretty well, so if you  
qualify, I'd encourage you to get involved. See http://puck.nether.net/mailman/listinfo/nsp-security 
  for info.

The NSP-SEC bof at NANOG is not quite the same environment as the NSP- 
SEC mailing list, but it generally includes the same people, plus  
others from the operations community who take the effort to attend  
NANOG, and so are sort of self-selected as being "one of the  
operators" with an already working amount of clue about the subjects  
that are being discussed. Additionally,  the concept of a "trusted  
environment" still sorta applies. You may not have realized it, but  
unlike all other sessions at NANOG, the slides are not published, they  
are not available online, and the session is not broadcast. So  
"Confidential" was there to remind folks in the BoF that this was a  
non-public (for a skewed version of public) presentation.

Having explained that bit of history which gives you a general  
background, let me deal with some specifics.

>
>
> Perhaps that is my own perspective (& paranoia?), but I found the CERT
> gentleman's call to monitor icmp backscatter on our authoritative
> nameservers far more informative -- and open.

I don't think anyone from CERT presented. Perhaps you meant Barry  
Green from Juniper's CERT team? Another "vendor"? Well, as you'll see  
further on, not really. In this context, like everyone else who  
presented, he was there as an operator, sharing knowledge and  
experience. But I digress...

>
>
> But I was disappointed with two vendors and their presentations: the  
> first
> had the tactic of saying "DNSSEC is the actual solution" when asked  
> about
> why their product would be necessary...completely ignoring the fact  
> that
> their proprietary "interim solution" was by no means the only way to  
> prevent
> cache poisoning attacks.  Indeed, I would daresay it isn't the best,  
> either
> by a BCP perspective, or a cost analysis perspective.

While we may disagree on your last claim (and I actually have a few  
years of experience to help me argue my point), I specifically said  
there were a) solutions that solved part of the problem (switching to  
TCP, detecting and blocking cache poisoning attacks) and b) the right  
solutions like DLV and DNSSEC that will take some time to be deployed.  
And I then made sure everyone heard me when I said that we need to  
find an interim solution that can be deployed *now*, until DNSSEC  
exists in a useful footprint. I ignore *nothing*. If you have another  
solution that solves the same problems that has running code now,  
please share it with all of us. Remember, it has to scale, it has to  
solve all of the problems, and it has to be implementable across a  
range of levels of clue.

>
>
> To put a finer point on this, i should say that i found myself  
> discomforted
> by a presentation suggesting that I should put their proprietary  
> appliances
> between my recursive name servers & the Net, and I am grateful that  
> Mr.
> Vixie stood up and said that there are other ways of dealing with the
> problem.

Indeed. Read further.
>

> Fortunately, said vendor had a table at "beer and gear", so I was  
> able to
> talk with one of their representatives -- and learned that they have  
> just as
> much trouble with automatic detection of attacks designed to look  
> like a
> "slashdotting"...which cleared up the mystery as to why it wasn't on  
> the
> graphs.
>
> Because this is a real problem:  anybody, with sufficient knowledge &
> preparation can vandalize _anybody's_ network.  Showing me a graph  
> that ping
> floods happen all the time doesn't impress me -- what would impress  
> me is
> going over the actual methods, algorithms (and heuristics?) used in  
> these
> attack mitigation appliances.
>
> Because, the "best" attack mitigation appliance vendor would seem to  
> have
> 100% of their market, and thus, charge exhorbant prices for their
> product(s).  When I brought this up with Mr. Vendor, his first  
> reaction was
> to point out that the cost was less than a home-grown solution.   
> When I
> raised the question of open source software to do the same thing, his
> reaction was to ask:  "oh? who's going to write it?"
>
> And that right there would seem to be a bit of bravado, perhaps  
> fueled by a
> misunderstanding of the role that FOSS has played on the Net.
>

> Fortunately -- and again, I am grateful for this -- the ISC was  
> represented
> in the security BOF, presenting the SIE concept...as well as what
> applications _already exist_ to detect and mitigate various  
> attacks.  One
> demonstration that blew me away:  detecting a botnet being set up  
> for a
> phishing attack...and preventing the attack before it even started.

Cool. I'm glad you saw value from that "vendor".
Seriously. SIE is good stuff.

>
>
> So in conclusion, I'll say this:  the last NANOG I attended was  
> NANOG 9 --
> and i remember that being a more challenging environment for vendors.
> Probably the biggest problem discussed back then was head-of-line  
> blocking
> on a vendor's switches.  _That_ is the kind of content that i have  
> found
> valuable, both on this list, and at a conference.
>
> And so:  If I weren't so knock-kneed in public venues,
> I would probably be doing what i would like to call on conference
> participants to do:  if someone gives a presentation that includes  
> their own
> proprietary black-box "solution", I think the best benefit for NANOG  
> would
> be to point out alternatives.

*I* was the "vendor" at the security BOF you took aim at. Except I am  
not a vendor in this environment. I am an operator. Just like ISC  
(Vixie) and McPherson (Arbor) and Greene (Juniper) etc. We are there  
as operators and *none* of us was selling *anything. We were  
describing issues that we currently are facing as operators, and  
solutions we have developed. You're not alone amongst "newcomers" in  
missing the point, so don't be hard on yourself ;-). In my case,  
*nothing* was being sold, other than *a* solution, which I am actually  
*giving* away to networks that matter in solving the probelm, and  
picking up the costs myself. I assume you missed that. And the reason  
I was doing that with a *proprietary* solution was because the open  
source solution is *not yet ready* for prime time, mainly because it  
(they) have not solved the wide implementation challenge. And *we*  
need to find a solution today while the open source (and best  
solution) gets rolled out effectively. Paul (also a "vendor" in the  
same vein, but an operator in the BoF forum) answered the question of  
whether there was another solution by saying "there is in Bind 9.6" -  
his product, which was released a couple of weeks ago.

I referred to it in my presentation, as a solution, along with  
DNSSEC.  It's called DLV. Unfortunately, and Paul admits it, there are  
challenges to widespread adoption. It works, but there is no business  
case that makes it easy to roll out. And therein lies the challenge.  
My customers need it today. And if it isn't out there in wide use, *it  
doesn't solve the problem*. So I am solving that by picking up the tab  
myself, and being reimbursed by the people I am a vendor to, my  
customers. And they're happy to pay for it. None of them were at the  
bof. Well, not strictly true, but not in numbers to matter. But  
hopefully you get the point. And you now understand that in the BoF we  
are all working to try and *solve* problems, not sell products. I'm  
sorry you failed to grok that difference.

Finally,  despite your knocking knees, you should have stood up and  
questioned anything you heard, or misunderstood. Then you would have  
had a better experience of the bof. As a member of the Program  
Committee and coincidentally the host of this NANOG, I'm sorry we  
didn't do a better job. We're trying to get better. I think that this  
was one of the best NANOGs we've ever had. But I'm biased, especially  
this time ;-).

As an aside, since you were last at a NANOG, we now have Beer 'n Gear,  
where Vendors have the opportunity to show off their wares, and in  
exchange they support and underwrite some of the costs of what is a  
pretty slick conference. I'm not sure why you believe that the vendor  
pitching his/her products at Beer 'n Gear is in some way violating the  
sacred rule against talking about a product. The B&G specifically  
provides the controlled environment and tradeoff. And *most* operators  
appreciate it, and make really good use of the opportunity to learn  
about new products that actually matter in such a useful environment.  
In one place we get to talk to actual engineers, about their products,  
together with 500 fellow operators who ask questions we may not even  
know we should ask.

If you have any other questions about my presentation, or the program,  
please feel free to ask directly.

>
>
> -Scott
> p.s. sorry for the long post.

Ditto for the response. But I have to assume you were not the only one  
who may have missed key points. Thanks for coming back. Hopefully  
we'll see you in the Dominican Republic next January.