The DDOS problem & security BOF: Am i mistaken?

Wed Oct 15 20:05:06 UTC 2008

On Oct 14, 2008, at 12:08 PM, Scott Doty wrote:

> First, the good news:  so far, the NANOG conference has been very  
> valuable and
> content-rich, covering a lot of issues that need to be discussed.   
> For that, I am grateful.
>
> But now, the bad news(?):  Maybe it's just me & my paranoia, but do  
> I detect
> an inkling of "murk spam" going on with some presentations?

I fully agree with you -- some talks are thinly (or not so thinly)  
veiled attempts to convince you to buy a vendor's shiny, new solution.  
There are a large number of reasons for this, and the Program  
Committee works hard (and I think is doing a great job) to limit the  
amount of sales pitch but A: there are a limited number of talks and  
B: many vendors are unable to resist trying to spin their product. I  
suggest that if you have a topic that you would like to present (and  
will keep it sales free) you resent it to the PC.

I *do* however disagree with you that this happened in the talks to  
which you are referring...

>
>
> Because there seems to be a fundamental misunderstanding, either on  
> my part,
> or the part of certain vendors: I'm hear to discuss ideas & freely  
> share
> them, and they are here to discuss (it would seem) their products.

Once again, great -- please submit a talk to the PC and they will  
review it. The PC is always looking for good talks...

> Sometimes
> both goals coincide, and that is fine...but...
>
> When a vendor at the security BOF starts showing documents that are  
> "company
> confidential", and trying to whip up a climate of fear, that we  
> should all
> deploy their product in front of our recursive name servers, i get  
> this
> funny feeling that I am being "murk spammed".

Hmmm... The vendor that you are referring to provides authoritative  
DNS for many domains (and, at least some of them I view as  
"important", meaning that I would prefer a correct response!). Yes, I  
am sure that he would be happy to have you as a customer and, yes,  
this is feature that differentiates his company, but I did not get the  
impression AT ALL that he was trying to sell his service, but rather  
provide better service to his existing customers, even going so far as  
to provide free devices to people who run large recursive resolvers.  
This helps both his existing customers (who, yes, will be more likely  
to continue using him), but, more importantly helps me as an end user  
feel a little comfortable that the page that I am getting is the  
correct page...

>
>
> Perhaps that is my own perspective (& paranoia?), but I found the CERT
> gentleman's call to monitor icmp backscatter on our authoritative
> nameservers far more informative -- and open.
>
> But I was disappointed with two vendors and their presentations: the  
> first
> had the tactic of saying "DNSSEC is the actual solution" when asked  
> about
> why their product would be necessary...completely ignoring the fact  
> that
> their proprietary "interim solution" was by no means the only way to  
> prevent
> cache poisoning attacks.

I may be mistaken, but I didn't get the impression that he believed  
that his solution was the only one -- he repeatedly pointed out that  
DNSSEC is the correct solution and this his solution does not solve  
all of the problems that DNSSEC would -- however, DNSSEC is FAR from  
being fully deployed.

>  Indeed, I would daresay it isn't the best, either
> by a BCP perspective, or a cost analysis perspective.

>
>
> To put a finer point on this, i should say that i found myself  
> discomforted
> by a presentation suggesting that I should put their proprietary  
> appliances
> between my recursive name servers & the Net, and I am grateful that  
> Mr.
> Vixie stood up and said that there are other ways of dealing with the
> problem.
>

Hmmm.. We must have VERY different recollections -- I don't remember  
him mentioning how much this would cost, other than that he would be  
give away some to the biggest wins first. Without knowing how much  
these widgets will be, it is not possibly to do a cost comparison, but  
don't discount just how expensive engineering time is, and just how  
hard it is to find competent DNS folks able to deploy something else.

I have chatted with many people about the state of their DNS  
infrastructure -- many people don't care, many people DO care but just  
don't have the cycles to properly maintain it, many have weird  
internal politics around them, and many just don't have the knowledge.  
Some of these are hard to solve, the lack of knowledge is probably the  
easiest, so I would welcome any how0-to, etc guides that would feel  
like writing....

> Then there was the gentleman with the DDOS detection/mitigation  
> appliance,
> who flipped through several graphs, which were intended to show the  
> number
> of each type of attack.  It's unfortunate that there wasn't more  
> time for
> questions, because I really wanted to ask why "http GET" and  
> "spidering"
> attacks weren't listen on their graphs...more on that in a second.
>

Hmmm, probably some of this is my fault, I am largely responsible for  
the agenda -- this was my first tie doing this an I suspect that I  
tried to fit too many talks into too little time. If there had been  
more time Danny might have covered their collection methodology (but,  
I need to warn you that that would probably have involved some  
information that *could* be construed as "This is what differentiates  
us" and would have been construed as sales, but whatever...). The  
information that was presented is part of a very well know report that  
gets published (but in a more executive format) and he (apparently  
incorrectly) assumed that the BOF audience would already be aware of  
how the information is collected and some of the benefits and short  
comings of their collection methodology. Once agin, probably my fault  
that he didn't have enough time to go though how the data is  
collected, but if he had, most of the audience would have bored out of  
their minds and they already know this and the rest would have felt  
like they were being sold to...

> Fortunately, said vendor had a table at "beer and gear", so I was  
> able to
> talk with one of their representatives -- and learned that they have  
> just as
> much trouble with automatic detection of attacks designed to look  
> like a
> "slashdotting"...which cleared up the mystery as to why it wasn't on  
> the
> graphs.
>

> Because this is a real problem:  anybody, with sufficient knowledge &
> preparation can vandalize _anybody's_ network.  Showing me a graph  
> that ping
> floods happen all the time doesn't impress me -- what would impress  
> me is
> going over the actual methods, algorithms (and heuristics?) used in  
> these
> attack mitigation appliances.

Ok, now I am confused --- you would like the vendor to stand up (in a  
NANOG presentation) and say: "Here is our widget, look how shiny it  
is.. Our device is better than $COMPETITOR because we do X, Y, Z, etc.  
We use the following heuristics <cough> and other vendors don't </ 
cough>"? To me this sound WAY more like a sales ploy (and, some of the  
other talks were much closer to this....).

>
>
> Because, the "best" attack mitigation appliance vendor would seem to  
> have
> 100% of their market, and thus, charge exhorbant prices for their
> product(s).  When I brought this up with Mr. Vendor, his first  
> reaction was
> to point out that the cost was less than a home-grown solution.

Yup... Said vendor does have a large market share -- by explaining how  
they collect the information they would have had to explain just how  
much of the Internet they instrument, which to me would have felt very  
salesey...

>  When I
> raised the question of open source software to do the same thing, his
> reaction was to ask:  "oh? who's going to write it?"
> And that right there would seem to be a bit of bravado, perhaps  
> fueled by a
> misunderstanding of the role that FOSS has played on the Net.

Yes, you can build your own attack mitigation solution (either based  
on OSS and / or from scratch), but there are limitations. Just saying  
"use OSS" doesn't make a fully formed solution spring into being,  
there are *large* investments needed in terms of time, effort,  
resource, scaling, training, lack of support, etc. While you *can*  
build a router using just OSS tools[0] there is a reason that most  
don't...

>
>
> Fortunately -- and again, I am grateful for this -- the ISC was  
> represented
> in the security BOF, presenting the SIE concept...as well as what
> applications _already exist_ to detect and mitigate various  
> attacks.  One
> demonstration that blew me away:  detecting a botnet being set up  
> for a
> phishing attack...and preventing the attack before it even started.

Great, I'm glad you liked that...

>
>
> So in conclusion, I'll say this:  the last NANOG I attended was  
> NANOG 9 --
> and i remember that being a more challenging environment for vendors.
> Probably the biggest problem discussed back then was head-of-line  
> blocking
> on a vendor's switches.  _That_ is the kind of content that i have  
> found
> valuable, both on this list, and at a conference.
>

Hmmm, I remember some of these -- and I remember the "Our box does  
this way better than $OTHER_VENDOR" spin that was always put on this...

> And so:  If I weren't so knock-kneed in public venues,
> I would probably be doing what i would like to call on conference
> participants to do:  if someone gives a presentation that includes  
> their own
> proprietary black-box "solution", I think the best benefit for NANOG  
> would
> be to point out alternatives.
>

Next time, please try and overcome your fear (although, I will happily  
point out that I haven't  -- even saying "sorry, only time for 1 more  
question" gives me sweaty palms, makes me feel queasy, etc. What helps  
is to remember just how badly most of the other people here speak and  
that no-one cares) -- other (sane and realistic) solutions are always  
welcomed...

> -Scott
> p.s. sorry for the long post.

W

[0]: OMG, have I just kicked off the "Liinux / BSD as your core  
router" discussion again?!