Amazon network engineering contact? re: DDoS traffic

Fri Nov 9 01:40:29 UTC 2018

Nobody should ever be forced to peer to get someone to address abusive
traffic originating from networks under their control.

On Thu, Nov 8, 2018 at 4:29 PM John <jw at nuclearfallout.net> wrote:

> Zach,
>
> As mentioned before, I am open to peering (where possible) but have not
> received a response.
>
> My goal is to connect with someone at Amazon and work with them on a
> technical solution, which is why I posted asking for that here. Various
> factors mean that we can't just upgrade our way out of this one, and
> manual whac-a-mole on the sources has shown to have limited use.
>
> These attacks also impact Amazon and the networks in between the sources
> and targets, and they take time to handle by the abuse teams, so there
> are good reasons to investigate them further and find better ways to
> mitigate or prevent them.
>
> -John
>
> On 11/8/2018 1:12 PM, Zach Puls wrote:
> > Makes sense, that's understandable. Do you peer with AWS? If not, maybe
> opening up a peering agreement will give you a better contact, and a bit
> more pull when attacks occur? I know someone with a peering agreement with
> AWS, and they have been able to get resolutions fairly quickly when issues
> arise.
> >
> > Other than that, I'm not sure of a solution other than more IP transit.
> >
> > Thanks,
> >
> > Zach Puls
> > Network Engineer | MEF-CECP
> > KsFiberNet
> >
> > -----Original Message-----
> > From: John Weekes <jw at nuclearfallout.net>
> > Sent: Thursday, November 08, 2018 15:03
> > To: Zach Puls <zpuls at ksfiber.net>
> > Cc: nanog at nanog.org
> > Subject: Re: Amazon network engineering contact? re: DDoS traffic
> >
> > Zach,
> >
> > Yes, RTBH is used to distribute the null-routes that I mentioned.
> >
> > Unfortunately, even brief saturation events lasting just 5-10 seconds (a
> typical amount of time to detect the loss, issue the null-route, and see
> the traffic start to fall off as it is distributed upstream) can cause real
> damage to those customers who are sensitive to latency and packet loss. So
> while null-routes limit the duration of the impact, they can't eliminate it
> entirely. And, of course, the actual target of the attack
> > -- the now-null-routed IP address -- becomes unreachable, which was
> presumably the goal of the attacker.
> >
> > -John
> >
> > On 11/8/2018 12:54 PM, Zach Puls wrote:
> >> No idea about an Amazon abuse contact, but do you have RTBH communities
> enabled with your upstream provider(s)? As a general practice, when you
> detect a (D)DoS attack in progress, it would help to automatically
> advertise that prefix to your upstream(s) with the black-hole community.
> This would at least help mitigate the effects of the attacks when they do
> occur, even if they come from a different source than AWS.
> >>
> >> Thanks,
> >>
> >> Zach Puls
> >> Network Engineer | MEF-CECP
> >> KsFiberNet
> >>
> >> -----Original Message-----
> >> From: NANOG <nanog-bounces at nanog.org> On Behalf Of John Weekes
> >> Sent: Thursday, November 08, 2018 14:44
> >> To: nanog at nanog.org
> >> Subject: Amazon network engineering contact? re: DDoS traffic
> >>
> >> We've been seeing significant attack activity from Amazon over the last
> two months, involving apparently compromised instances that commonly send
> 1-10G of traffic per source and together generate Nx10G of total traffic.
> Even when our overall upstream capacity exceeds an attack's overall size,
> the nature of load-balancing over multiple 10G upstream links means that an
> individual link can be saturated by multiple large flows, forcing our
> systems to null-route the target to limit impact.
> >>
> >> We've sent an abuse notification about every traffic source to Amazon,
> and specific sources seem to stop their involvement over time (suggesting
> that abuse teams are following up on them), but there is an endless parade
> of new attackers, and each source participates in many damaging attacks
> before it is shut down.
> >>
> >> Is there anyone at Amazon who can help with an engineering solution in
> terms of programmatically detecting and rate-limiting attack traffic
> sources, to our networks or overall? Or applying the kludge of a rate-limit
> for all Amazon traffic to our networks? Or working with us on some other
> option?
> >>
> >> At least one other large cloud provider has an automatic rate-limiting
> system in place that is effective in reducing the damage from repeat
> high-volume attacks.
> >>
> >> Emails to the Amazon NOC, peering contacts (since that would be another
> possible solution), and abuse department have not connected me with anyone.
> >>
> >> Thanks,
> >> John
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20181108/6ff5997f/attachment.html>