The DDOS problem & security BOF: Am i mistaken?
rjoffe at centergate.com
Wed Oct 15 15:39:58 CDT 2008
On Oct 14, 2008, at 9:08 AM, Scott Doty wrote:
> First, the good news: so far, the NANOG conference has been very
> valuable and
> content-rich, covering a lot of issues that need to be discussed.
> For that, I am grateful.
Thank you. We worked hard to make it valuable.
> But now, the bad news(?): Maybe it's just me & my paranoia, but do
> I detect
> an inkling of "murk spam" going on with some presentations?
Not sure what you mean by "murk spam". Thats a term that died years
ago. And it really related to people claiming that spam was "in
compliance with federal laws". But I think I can guess your intentions
from the tone of your email, so let me try and respond.
> Because there seems to be a fundamental misunderstanding, either on
> my part,
> or the part of certain vendors: I'm hear to discuss ideas & freely
> them, and they are here to discuss (it would seem) their products.
> both goals coincide, and that is fine...but...
> When a vendor at the security BOF starts showing documents that are
> confidential", and trying to whip up a climate of fear, that we
> should all
> deploy their product in front of our recursive name servers, i get
> funny feeling that I am being "murk spammed".
Well, that's interesting. I see your last NANOG was 9, in February of
1997. So "Welcome back!". We're glad to have you here in person.
Things have changed slightly since then. NSP-SEC never existed in
1997. It really came about in the early 2000's where it was developed
as a forum for actual operators to share views and thoughts, generally
in real time, to help the 'net in general survive disruption,
malicious or otherwise. It has really worked pretty well, so if you
qualify, I'd encourage you to get involved. See http://puck.nether.net/mailman/listinfo/nsp-security
The NSP-SEC bof at NANOG is not quite the same environment as the NSP-
SEC mailing list, but it generally includes the same people, plus
others from the operations community who take the effort to attend
NANOG, and so are sort of self-selected as being "one of the
operators" with an already working amount of clue about the subjects
that are being discussed. Additionally, the concept of a "trusted
environment" still sorta applies. You may not have realized it, but
unlike all other sessions at NANOG, the slides are not published, they
are not available online, and the session is not broadcast. So
"Confidential" was there to remind folks in the BoF that this was a
non-public (for a skewed version of public) presentation.
Having explained that bit of history which gives you a general
background, let me deal with some specifics.
> Perhaps that is my own perspective (& paranoia?), but I found the CERT
> gentleman's call to monitor icmp backscatter on our authoritative
> nameservers far more informative -- and open.
I don't think anyone from CERT presented. Perhaps you meant Barry
Green from Juniper's CERT team? Another "vendor"? Well, as you'll see
further on, not really. In this context, like everyone else who
presented, he was there as an operator, sharing knowledge and
experience. But I digress...
> But I was disappointed with two vendors and their presentations: the
> had the tactic of saying "DNSSEC is the actual solution" when asked
> why their product would be necessary...completely ignoring the fact
> their proprietary "interim solution" was by no means the only way to
> cache poisoning attacks. Indeed, I would daresay it isn't the best,
> by a BCP perspective, or a cost analysis perspective.
While we may disagree on your last claim (and I actually have a few
years of experience to help me argue my point), I specifically said
there were a) solutions that solved part of the problem (switching to
TCP, detecting and blocking cache poisoning attacks) and b) the right
solutions like DLV and DNSSEC that will take some time to be deployed.
And I then made sure everyone heard me when I said that we need to
find an interim solution that can be deployed *now*, until DNSSEC
exists in a useful footprint. I ignore *nothing*. If you have another
solution that solves the same problems that has running code now,
please share it with all of us. Remember, it has to scale, it has to
solve all of the problems, and it has to be implementable across a
range of levels of clue.
> To put a finer point on this, i should say that i found myself
> by a presentation suggesting that I should put their proprietary
> between my recursive name servers & the Net, and I am grateful that
> Vixie stood up and said that there are other ways of dealing with the
Indeed. Read further.
> Fortunately, said vendor had a table at "beer and gear", so I was
> able to
> talk with one of their representatives -- and learned that they have
> just as
> much trouble with automatic detection of attacks designed to look
> like a
> "slashdotting"...which cleared up the mystery as to why it wasn't on
> Because this is a real problem: anybody, with sufficient knowledge &
> preparation can vandalize _anybody's_ network. Showing me a graph
> that ping
> floods happen all the time doesn't impress me -- what would impress
> me is
> going over the actual methods, algorithms (and heuristics?) used in
> attack mitigation appliances.
> Because, the "best" attack mitigation appliance vendor would seem to
> 100% of their market, and thus, charge exhorbant prices for their
> product(s). When I brought this up with Mr. Vendor, his first
> reaction was
> to point out that the cost was less than a home-grown solution.
> When I
> raised the question of open source software to do the same thing, his
> reaction was to ask: "oh? who's going to write it?"
> And that right there would seem to be a bit of bravado, perhaps
> fueled by a
> misunderstanding of the role that FOSS has played on the Net.
> Fortunately -- and again, I am grateful for this -- the ISC was
> in the security BOF, presenting the SIE concept...as well as what
> applications _already exist_ to detect and mitigate various
> attacks. One
> demonstration that blew me away: detecting a botnet being set up
> for a
> phishing attack...and preventing the attack before it even started.
Cool. I'm glad you saw value from that "vendor".
Seriously. SIE is good stuff.
> So in conclusion, I'll say this: the last NANOG I attended was
> NANOG 9 --
> and i remember that being a more challenging environment for vendors.
> Probably the biggest problem discussed back then was head-of-line
> on a vendor's switches. _That_ is the kind of content that i have
> valuable, both on this list, and at a conference.
> And so: If I weren't so knock-kneed in public venues,
> I would probably be doing what i would like to call on conference
> participants to do: if someone gives a presentation that includes
> their own
> proprietary black-box "solution", I think the best benefit for NANOG
> be to point out alternatives.
*I* was the "vendor" at the security BOF you took aim at. Except I am
not a vendor in this environment. I am an operator. Just like ISC
(Vixie) and McPherson (Arbor) and Greene (Juniper) etc. We are there
as operators and *none* of us was selling *anything. We were
describing issues that we currently are facing as operators, and
solutions we have developed. You're not alone amongst "newcomers" in
missing the point, so don't be hard on yourself ;-). In my case,
*nothing* was being sold, other than *a* solution, which I am actually
*giving* away to networks that matter in solving the probelm, and
picking up the costs myself. I assume you missed that. And the reason
I was doing that with a *proprietary* solution was because the open
source solution is *not yet ready* for prime time, mainly because it
(they) have not solved the wide implementation challenge. And *we*
need to find a solution today while the open source (and best
solution) gets rolled out effectively. Paul (also a "vendor" in the
same vein, but an operator in the BoF forum) answered the question of
whether there was another solution by saying "there is in Bind 9.6" -
his product, which was released a couple of weeks ago.
I referred to it in my presentation, as a solution, along with
DNSSEC. It's called DLV. Unfortunately, and Paul admits it, there are
challenges to widespread adoption. It works, but there is no business
case that makes it easy to roll out. And therein lies the challenge.
My customers need it today. And if it isn't out there in wide use, *it
doesn't solve the problem*. So I am solving that by picking up the tab
myself, and being reimbursed by the people I am a vendor to, my
customers. And they're happy to pay for it. None of them were at the
bof. Well, not strictly true, but not in numbers to matter. But
hopefully you get the point. And you now understand that in the BoF we
are all working to try and *solve* problems, not sell products. I'm
sorry you failed to grok that difference.
Finally, despite your knocking knees, you should have stood up and
questioned anything you heard, or misunderstood. Then you would have
had a better experience of the bof. As a member of the Program
Committee and coincidentally the host of this NANOG, I'm sorry we
didn't do a better job. We're trying to get better. I think that this
was one of the best NANOGs we've ever had. But I'm biased, especially
this time ;-).
As an aside, since you were last at a NANOG, we now have Beer 'n Gear,
where Vendors have the opportunity to show off their wares, and in
exchange they support and underwrite some of the costs of what is a
pretty slick conference. I'm not sure why you believe that the vendor
pitching his/her products at Beer 'n Gear is in some way violating the
sacred rule against talking about a product. The B&G specifically
provides the controlled environment and tradeoff. And *most* operators
appreciate it, and make really good use of the opportunity to learn
about new products that actually matter in such a useful environment.
In one place we get to talk to actual engineers, about their products,
together with 500 fellow operators who ask questions we may not even
know we should ask.
If you have any other questions about my presentation, or the program,
please feel free to ask directly.
> p.s. sorry for the long post.
Ditto for the response. But I have to assume you were not the only one
who may have missed key points. Thanks for coming back. Hopefully
we'll see you in the Dominican Republic next January.
More information about the NANOG