backtracking forged packets?

Mon Mar 16 14:24:24 UTC 2020

On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote:
> I don't recommend filtering the SYN-ACK packets.  That's what Octolus did,
> and the result was leaving half-open SYN_RECV connections on all the nodes
> used for reflection.  That has two downsides:
>
>   - the reflectors will retry the SYN-ACK (several times), which increases
> your PPS load (amplifying the attack)
>   - the providers may notice the large number of SYN_RECV connections from
> your network and put you on a blacklist

I work at Path Network; we're providing DDoS mitigation services for
Octolus. I wanted to follow up your message with a couple of points from
our POV.

The filter we have running for Octolus is a more generic stateful TCP
filter, designed primarily to curb spoofed TCP floods. The bigger
attacks we see are on the magnitude of 100s of Gbps/Mpps, and a simple
fact is that outright dropping an out-of-state packet is multiple times
less expensive for us than creating and responding with a RST.

We see spoofed TCP attacks much more frequently than a SYN-ACK
reflection, and in fact from our automation's point of view this looks
exactly like a SYN-ACK flood from randomized source addresses. Aside
from the resources cost, sending back RSTs for non-reflected attacks of
respectable volume might also not be appreciated by some networks.

Knowing the specifics of this long-running reflection attacks, we're
considering how to reply with RSTs so to not leave lingering SYN_RECVs
on reflectors' side.

On 2020-03-15 16:36, Jean | ddostest.me <jean at ddostest.me> wrote:
> I believe that Oculus blocked the RST and not the SYN/ACK.

The SYN-ACKs are dropped; letting them reach the end servers and
dropping outgoing RSTs instead would make for poor mitigation. :)

On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote (cont.):
> I don't want to leave you with the impression that it's hopeless... these
> attacks aren't impossible to stop --- it just requires convincing the
> transit providers to care.
>
> Damian
>
> On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG <
> nanog at nanog.org> wrote:
>
> Hi Bill,
>
> thanks for sharing the data. Indeed, I can't offer you a way to backtrack
> the spoofed packets.
>
> Anyway, I'm not sure what could you do legally as there is a very high
> chance that these people are not in the USA and the CFAA won't apply to
> them.
>
> Here is what I would do if I was in your situation.
>
> Since these packets are spoof and malformed, I would block all SYN/ACK
> based on the length.
>
> Depending on your hardware, it's very easy to inspect *only the SYN/ACK
> by length* if you have modern hardware. On linux/unix, it's also very
> straightforward. I'm not sure for windows though.
>
> Here is the details of the analysis:
>
> Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS,
> SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be
> others, but let's use the basic one.
>
> In your case, there are none. There is only MSS and the SYN length is 44
> bytes. These are spoof packets maybe generated by either TCP-AMP like
> reported earlier.
>
> I would try to block all SYN/ACK coming toward your network with a length
> of 44 bytes or lower. But, this is weird because it should be 54 bytes.
> Maybe there is some offloading of some sort in your gear.
>
> Now depending on your hardware, it could work or it could kill your router
> as it will punt the cpu. I guess you have some modern gear.
>
> What I do when I am not sure about the length, I start to accept and log
> at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins.
>
> Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower
> than X bytes. It won't kill or block anything legitimate if you do it
> properly. :)
>
> What will happen is that you will not reply to these spoof SYN/ACK with a
> RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your
> service providers will be happy and should not penalize you.
>
> I'm pretty sure that it will help you as it did for me in the past.
>
> Let me know if it's not clear and/or which part is foggy and I'll try to
> give more details and better explanation.
>
> Regards,
>
> Jean St-Laurent
> On 2020-03-14 11:46, William Herrin wrote:
>
> On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog at
nanog.org> <nanog at nanog.org> wrote:
>
> can you post some forged packets please? You can send them offlist if
> you prefer.
>
> Hi Jean,
>
> Here are a couple examples (PDT this morning):
>
> 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none],
> proto ICMP (1), length 56)
>     45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable -
> admin prohibited filter, length 36
>         IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP
> (6), length 40)
>     199.33.225.218.9851 > 45.89.93.26.443: [|tcp]
>         0x0000:  4500 0038 277c 0000 3701 28da 2d59 5d1a
>         0x0010:  c721 e1da 030d 4b61 0000 0000 4500 0028
>         0x0020:  277c 4000 4506 dae4 c721 e1da 2d59 5d1a
>         0x0030:  267b 01bb a057 e903
>
> 08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto
> TCP (6), length 44)
>     104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a
> (correct), seq 1216155085, ack 11765035, win 29200, options [mss
> 1156], length 0
>         0x0000:  4500 002c 0000 4000 3606 e564 6857 4e5f
>         0x0010:  c721 e18f 0050 21db 487d 0dcd 00b3 852b
>         0x0020:  6012 7210 c97a 0000 0204 0484
>
> I have observed no consistency in the remote IP addresses. I receive
> no more than a few of each and they don't line up with particular
> networks. Remote ports are heavily 80, 443, 22, 25, etc. but a
> smattering of less common ports too. I'm not seeing any RSTs at all
> nor any port-unreachables. Lots of syn/acks and a few time exceeded
> and host unreachables. I don't know what to make of that.
>
>
> On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith at
gmail.com> <andrew.william.smith at gmail.com> wrote:
>
> Look inside the ICMP Unreachable backscatter at the truncated original
packet that caused the unreachable message.
>
> Clever! I wouldn't have thought of that. Unfortunately as in the
> example above, the TTLs in the packets encapsulated in ICMP are not
> especially close to one of the common boundaries.
>
> Regards,
> Bill Herrin
>
> --
> William Herrinbill at herrin.ushttps://bill.herrin.us/
>
>