Death of the Internet, Film at 11

Josh Reynolds josh at
Mon Oct 24 12:03:45 UTC 2016

You CAN actually block things, within reason. The caveat is you simply have
to disclose it. There is a 'reasonable network management' clause. IANAL,
please consult your telecommunications legal team.

On Oct 24, 2016 1:25 AM, "Richard Holbo" <holbor at> wrote:

> I run/manage the networks for several smallish (in the thousands of
> customers) eyeball ISP's and  I appreciate a nice "hey you've got a bot" or
> "someone is scanning" me notice to my abuse emails.  They are useful in
> identifying crap that's going on, so for those of you who have the
> resources to do that...  I appreciate it, we do read them at my networks
> and try to do something.
> That said... getting end users to actually fix the broken routers etc. etc.
> is NOT easy.    Very often we'll notify customers, they will _take their
> stuff to the local computer repair guy_ ... or office depo.... and they
> will run whatever auto scan they have and say it's all fine.  Customer puts
> it back in, it's still broke, and they call customer support and want us to
> pay for the trip because _their_ expert says it's fine...
> IMHO since the advent of Net Neutrality... I cannot simply block all of X,
> Y or Z at my edge and tell the customers it's for the best.  I'd love to
> block some stuff in and outbound to customers, but then the customer just
> yells at us and files complaints with the PUC because _they have a right to
> it_.. So those of you calling for Government interference... we've already
> done that and it does not help.
> /rh
> On Sun, Oct 23, 2016 at 10:56 PM, John Weekes <jw at>
> wrote:
> > On 10/23/2016 4:19 PM, Ronald F. Guilmette wrote:
> >
> >>
> >> ... I've recorded
> >>> about 2.4 million IP addresses involved in the last two months (a
> number
> >>> that is higher than the number of actual devices, since most seem to
> >>> have dynamic IP addresses). The ISPs behind those IP addresses have
> >>> received notifications via email...
> >>>
> >> Just curious... How well is that working out?
> >>
> >
> > For the IoT botnets, most of the emails are ignored or rejected, because
> > most go to providers who either quietly bitbucket them or flat-out reject
> > all abuse emails. Most emails sent to mainland China, for instance, are
> in
> > that category (Hong Kong ISPs are somewhat better).
> >
> > For other botnets, such as those using compromised webservers running
> > outdated phpMyAdmin installs at random hosts, harnessing spun-up services
> > at reputable VPS providers (Amazon, Microsoft, Rackspace, etc.), or
> > harnessing devices at large and small US and Canadian ISPs, we have had
> > better luck. Usually, we don't hear a response back, but those emails are
> > often forwarded to the end-user, who takes action (and may ask us for
> help,
> > which is how we know they are being forwarded). The fixes can enough to
> > reduce attack volumes to more manageable levels.
> >
> > Kudos go out to the large and small ISPs and NSPs who have started
> > policing SSDP and other reflection traffic, which we also send out some
> > notifications for. In some cases, it may be that our emails spurred them
> to
> > notice how much damage those attacks were doing and how much it was
> costing
> > them to carry the attack traffic.
> >
> > I've tried this myself a few times in the past, when I've found things
> >> that appear to be seriously compromised, and for my extensive trouble
> >> I've mostly received back utter silence and no action.  I remember that
> >> after properly notifying [email protected] some large end-luser cable network
> >> in the SouthEast (which shall remain nameless) I got back something
> >> along the lines of "Thank you.  We'll look into it." and was disgusted
> >> to find, two months later, that the boxes in question were still utterly
> >> pwned and in the exact same state they were two months prior, when I
> >> had first reported them.
> >>
> >
> > We do get our share of that, as well, unfortunately, along with our share
> > of people who send angry responses calling the notifications spam (I
> > disagree with them that sending a legitimate abuse notification to a
> > publicly-posted, designated abuse account should be considered spam) or
> who
> > flame us for acting like "internet police". But, we persist. Some people
> > change their minds after receiving multiple notifications or after we
> > explain that DoS traffic costs them money and hurts their customers, who
> > will be experiencing degraded service and may silently switch providers
> > over it.
> >
> > I guess that's just an example of what somebody else already noted here,
> >> i.e. that providers don't care to spend the time and/or effort and/or
> >> money necessary to actually -do- anything about compromised boxes, and
> >> anyway, they don't want to lose a paying customer.
> >>
> >> So, you know, let's just say for the sake of argument that right now,
> >> today, I know about a botnet consiting of a quarter million popped
> >> boxes, and that I have in-hand all of the relevant IPs, and that I
> >> have no trouble finding contact email addresses for all of the relevant
> >> ASNs.  So then what?
> >>
> >
> > I use scripts to send out an abuse notification to some percentage of the
> > compromised hosts -- the ones sending some significant amount of the
> > traffic. The notification includes a description of what we saw and
> > timestamped example attack traffic, as interpreted by tcpdump. If further
> > traffic is seen later from the same host, another notification will be
> > sent, after a cool-off period.
> >
> > The emails are plain text and we don't try to use them as advertisement.
> > We also don't force a link to be clicked to see more details or to
> respond
> > back. I don't like to receive such emails myself and have found that
> those
> > types are more likely to be ignored.
> >
> > The question is:  Why should I waste my time informing all, or even any
> >> of these ASNs about the popped boxes on their networks when (a) I am
> >> not their customer... as many of them have been only too happy to
> >> gleefully inform me in the past... and when (b) the vast majority
> >> simply won't do anything with the information?
> >>
> >
> > I'm not saying that everyone should send abuse notifications like we do,
> > since it can be a big task. But, in response to someone wondering if
> their
> > network is being used for attacks, or asking how they could help to
> police
> > their own network, I am saying that making sure that inbound abuse
> > notifications are arriving at the right place and being handled
> > appropriately is important.
> >
> > And while we are on the subject, I just have to bring up one of my
> >> biggest pet peeves.  Why is it that every time some public-spirited
> >> altrusitc well-meaning citizen such as myself reports any kind of a
> >> problem to any kind of a company on the Internet, the report itself
> >> gets immediately labeled and categorized as a "complaint".  If I spend
> >> some of -my- valuable time to helpfully try to let somebody else know
> >> of a problem on their network, or with their web site, and if that
> >> report gets categorized as a "complaint" then what does that make me?
> >> A "complainer"??
> >>
> >> I don't need this kind of abuse and denegration from people who I'm
> >> trying to help.  Like most other people, if I am in need of some
> >> personal denegration and abuse... well... I have relatives for that.
> >>
> >
> > There's a spectrum of people responding to these and some percentage are
> > just jerks, as in real life. But, I like to think that the majority of at
> > least NA providers are represented by professionals who just don't
> respond
> > out of courtesy because they don't want to flood our inboxes with simple
> > acknowledgements.
> >
> > Those of us experiencing these attacks appreciate the community support,
> > both from people like you who also send notifications and those who
> handle
> > the notifications on the receiving end.
> >
> > -John
> >

More information about the NANOG mailing list