Global Akamai Outage

Tue Jul 27 20:38:41 UTC 2021

On Tue, 27 Jul 2021 at 16:10, Mark Tinka <mark at tinka.africa> wrote:
>
>
>
> On 7/26/21 19:04, Lukas Tribus wrote:
>
> > rpki-client can only remove outdated VRP's, if it a) actually runs and
> > b) if it successfully completes a validation cycle. It also needs to
> > do this BEFORE the RTR server distributes data.
> >
> > If rpki-client for whatever reason doesn't complete a validation cycle
> > [doesn't start, crashes, cannot write to the file] it will not be able
> > to update the file, which stayrtr reads and distributes.
>
> Have you had an odd experiences with rpki-client running? The fact that
> it's not a daemon suggests that it is less likely to bomb out (even
> though that could happen as a runtime binary, but one can reliably test
> for that with any effected changes).

No, I did not have a specific negative experience running rpki-client.

I did have my fair share of:

- fat fingering cronjobs
- fat fingering permissions
- read-only filesystems due to storage/virtualizations problems
- longer VM downtimes

I was also directly impacted by a hung rpki-validator, which I have
referenced in one of the links earlier. This was actually after I
started to have concerns about the lack of monitoring and the danger
of serving stale data, not before.

I was also constantly impacted by generic (non RPKI related) gray
failures in other people's network for the better part of a decade, I
guess that makes me particularly sensitive to topics like this.

> Of course, rpki-client depends on Cron being available and stable, and
> over the years, I have not run into any major issues guaranteeing that.

It's not the quality of the cron code that I'm worried about. It's the
amount of variables that can cause rpki-client to not complete and
fully write the validation results to disk COMBINED with the lack of
monitoring.

You get an alert in your NMS when a link is down, even if that single
link down doesn't mean your customers are impacted. But you need to
know so that you can actually intervene to restore full redundancy.
Lack of awareness of a problem is the large issue here.

> > If your VM went down with both rpki-client and stayrtr, and it stays
> > down for 2 days (maybe a nasty storage or virtualization problem or
> > maybe this just a PSU failure in a SPOF server), when the VM comes
> > backup, stayrtr will read and distribute 2 days old data - after all -
> > rpki-client is a periodic cronjob while stayrtr will start
> > immediately, so there will be plenty of time to distribute obsolete
> > VRP's. Just because you have another validator and RTR server in
> > another region that was always available, doesn't mean that the
> > erroneous and obsolete data served by this server will be ignored.
>
> This is a good point.
>
> So I know that one of the developers of StayRTR is working on having it
> use the "expires" values that rpki-client inherently possesses to ensure
> that StayRTR never delivers stale data to clients. If this works, while
> it does not eliminate the need to some degree of monitoring, it
> certainly makes it less of a hassle, going forward.

Notice that expires is based on the cryptographic validity of the ROA
objects. It can be multiple DAYS until expiration strikes, for example
the expiration value of 8.8.8.0/24 is 2 DAYS in the future.

> > There are more reasons and failure scenarios why this 2 piece setup
> > (periodic RPKI validation, separate RTR daemon) can become a "split
> > brain". As you implement more complicated setups (a single global RPKI
> > validation result is distributed to regional RTR servers - the
> > cloudflare approach), things get even more complicated. Generally I
> > prefer the all in one approach for these reasons (FORT validator).
> >
> > At least if it crashes, it takes down the RTR server with it:
> >
> > https://github.com/NICMx/FORT-validator/issues/40#issuecomment-695054163
> >
> >
> > But I have to emphasize that all those are just examples. Unknown bugs
> > or corner cases can lead to similar behavior in "all in one" daemons
> > like Fort and Routinator. That's why specific improvements absolutely
> > do not mean we don't have to monitor the RTR servers.
>
> Agreed.
>
> I've had my fair share of Fort issues in the past month, all of which
> have been fixed and a new release is imminent, so I'm happy.
>
> I'm currently running both Fort and rpki-client + StayRTR. At a basic
> level, they both send the exact same number of VRP's toward clients,
> likely because they share a philosophy in validation schemes, and crypto
> libraries.
>
> We're getting there.

For IOS-XR I have a netconf script that makes all kinds of health
checks at XR RTR client level:

- comparing the number of total IPv4 and v6 VRP's of each RTR server
enable with absolute values, warning if there are less than EXPECTED
values
- comparing the v4 and v6 number between the RTR endpoints on this XR
box, warning if the disparity crosses a threshold
- warning if a configured RTR server is not in connected state

https://gist.github.com/lukastribus/695c9e780d118755271519d4d3cb54f3

This is also useful for RTR client bugs in XR, which I have seen (the
state machine is broken, you need to "clear bgp rkpi ... server XY" to
restore it). I have not seen this issue in newer code fortunately.

lukas