DoS Attacks

Wed Oct 8 03:45:35 UTC 2003

Oh boy, what a fun night this was.  After a 4 or so hours downtime, my
servers are back up and running.

Heres the gorey details.

At about 7pm EST, we began having unusual issues with our network, the
router, and several machines on the network.  For the first part of the
attack, we were held down for a good 30-60 minutes.  Took us a while to
figure out which one of our machines was being targeted.  Turns out to be
our NAT firewall box.  We tried several things to drop the attack, but it
still kept coming in strong (mind you, we don't have very much bandwidth,
but we can usually ride out DoS attacks pretty well - this was an exception)
Then suddenly, out of the blue it dropped.  Outside connectivity was
restored and things were back to normal.

20 minutes later, the relentless attack began again.  This time, we were
ready and waiting with tcpdump and several other handcrafted tools we use
for this type of thing.  The attack was coming from a single source machine,
unspoofed (ballsy if you ask me), 128.186.11.215.  Packets were UDP, random
from 2100-2299 source and 2400-2699 dest.

So, now for the fun part.  Being offsite, I wasn't the one to place the
calls, but my admin on site started with FSU's abuse desk.  No help
whatsoever.  Claimed that because the abuse desk was gone, they had no
authority to deal with the problem.  Frustrated, annoyed, and pissed off, he
tried again, and got hung up on twice.  Nice people eh?

Our next call was a bit later (at this point, we were very unhappy and ready
to start raising hell with anyone we could find) - this time, to their
upstream Qwest.  After dealing with the operator, they finally sent him to
one of the NOCs.  Unfortunately, they sent him to the wrong NOC and not the
Qwest MD NOC.  Luckily, we got someone with a clue - a nice guy by the name
of Richard Stein who tried to help us, but found that the other NOC was
unresponsive and couldn't do anything himself to solve the problem.

After hanging up with Qwest, we got a call back from FSU.  After a good 20
minutes or so of talking with the net admin from FSU, things were finally
set in motion.  After another good 10 minutes or so, connectivity was
restored and everything was back to normal.  According to my guy,  they
yanked the whole subnet at FSU.  Problem solved.

So here I am, asking if anyone here has any advice on dealing with these
issues in the future?  Its painfully apparent noone takes these situations
seriously enough.  What should we do when we are put in a position like
this?  Just sit back and hope it goes away itself?

Also, any ideas on how to deal with these attacks on lower bandwidth
connections?  Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256
business DSL line.  I really can't afford to be buying T1s or T3s just to
hold up to attacks like this.

As always, thanks.
--------------------------
Brian Bruns
The Summit Open Source Development Group
Open Solutions For A Closed World / Anti-Spam Resources
http://www.2mbit.com
ICQ: 8077511