Questions for Level3 & Choopa about their Enabling of IP Hijacking (RPKI Invalid)

Fri Dec 11 17:14:09 UTC 2020

Is there anyone on this list at Level3 or Choopa who can respond to why:
1. Level3/Centurylink/Lumen: How come RPKI invalid prefixes were allowed 
to be announced and considered valid?
2. Level3/Centurylink/Lumen: Is there any actual NOC hidden behind the 
numbers where someone can reasonably be of assistance? Why are your SOC, 
Support and DDoS team saying you can't blackhole a hijacked prefix 
because of "GOVERNMENT regulations"?
3. CHOOPA: Why does your Vultr brand allow anyone who can successfully 
insert an IRR record (eg, in RaDB) to be imported (even when RPKI is 
invalid) and permit hijacking?
4. CHOOPA: Why does your Network Team need 3 days to check an RPKI valid 
or invalid? It is not rocket science.

Please feel free to contact me off the list with these answers. If you 
are interested in the long story of the whole ordeal of being hijacked 
for 3 days, it is below.

IP hijacking ordeal we went through:

We are extremely disappointed with Choopa/Vultr and Level3. On
2020-12-07 at exactly 23:48:10, we notified Choopa and Vultr via email,
ticket and contact form that one of our IPv4 prefixes was being hijacked
by one of their customers. Our prefixes are RPKI signed, and the ASN
announcing the prefix was not in the RPKI sign.

We continued to follow up on the request, and sent more requests in to
Vultr/Choopa's system. When we phoned Choopa, we were told by the
individual on the phone that they see the ticket, and they'll bump it up
on the network engineering ticket list.

Frustrated after three days of an ongoing hijack, and consistently no
assistance from Vultr or it's parent Choopa, we reached out to the only
Tier-1 Choopa has in London that was NOT filtering according to RPKI,
Level3.

We sent an email to the noc at level3carrier.com, which was listed as a
point of contact on the ASN & PeeringDB pages. The email had no
response. We then called the number (1-833-453-8353), and spoke to
Technical Support, who transferred us the first time to the SOC, the
second time to their DDoS department.

The person who responded in the SOC said that they can filter it "very
fast" if we validate ownership of the prefix (despite being RPKI
signed). So we complied, we were told to email
"abuse at centurylinkservices.net" with a message saying the prefix is
being hijacked, the ASN of the hijacker and the direct upstream
(Vultr/Choopa). We sent that email, he said we "should get" an automatic
reply (none arrived, confirmed with mx it was delivered). We waited half
an hour, which they agreed was a "reasonable time" to wait for it to be
filtered.

After half an hour, we followed up, and this time landed on the DDoS
department (I have no clue how they thought this through). At the DDoS
department, they said that they can't help, and I should "keep emailing"
abuse at centurylinkservices.net. He offered a phone number for me to call
as a "direct line to abuse", upon hanging up and dialing, I got to the
generic prompts for *customer services*. They were no help either.

Fast forward four hours, we have no point of contact at Level3 or
Choopa, we have had no communication from either. We finally get a
message from Choopa, reading:

"Greetings,

This ticket has been forwarded to our networking team so they can
examine your situation, check the infrastructure configuration, and
apply any relevant changes. Please allow for significant additional time
while we review this ticket."

"Significant additional time" to check an RPKI valid? That seems
incredibly odd. Fast forward to another hour, we receive the next of
Choopa messages, this time saying: "Thanks for the update on this. We
have validated the removal request and have removed the prefix from our
network. Please allow additional time for this to update to the
providers.".

We finally thought we were in the clear. Almost two hours go by, and the
prefix is still not filtered what-so-ever. We follow up once more, and
are told "We have removed the prefix and is not announced by Vultr
anymore. Please allow 24 to 48 hours for the internet providers to
update their routing database."

Finally, an hour later, nlnog & other lg's are seeing Level3 no longer
announcing the prefix.

What we found out in the process is Vultr ignores RPKI invalid (despite
having a table on their system which shows RPKI Invalid/Invalid ASN), as
long as at one point in time an IRR record existed (or, is created).
Once Vultr gets their hands on it, they make the IRR records at RaDB,
and keep updating them even when they're not valid.

With the run around of Level3, and Choopa/Vultr, they're practically
inviting IP hijackers to play. Insanity!