afrinic rpki issue

Wed Jun 14 14:05:09 UTC 2023

Hi Carlos,

Because of the issues that AfriNIC is facing, they are forcing all traffic from HTTPS to rsync, so you should check if rsync can properly set up outbound connections from your machine. What’s the output you get when you rsync rsync://rpki.afrinic.net/repository/ ?

I do an interactive Routinator validation run with debug logging enabled, like so:

$ routinator -vv vrps -f summary

Then I see the following in the logs:

[WARN] RRDP https://rrdp.afrinic.net/notification.xml: Getting notification file failed with status 204 No Content
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Update failed and current copy is expired since 2023-05-30 10:43:44 UTC.
[INFO] RRDP https://rrdp.afrinic.net/notification.xml: Falling back to rsync.
[INFO] rsyncing from rsync://rpki.afrinic.net/repository/.

Then, rsyncing the contents works just fine; objects are fetched and validated. Some objects fail validation with "certificate is not yet valid.”, "certificate has been revoked.” and “Object not found.” but that appears unrelated to the connectivity issues they’re facing. 

I end up with the following totals:

Summary at 2023-06-14 13:43:24.366013 UTC
afrinic:              ROAs:    5756 verified;
            VRPs:    7121 verified,    6820 final;
    router certs:       0 verified;
     router keys:       0 verified,       0 final.
           ASPAs:       0 verified,       0 final.

If you want some logs to compare, you can have a look here:
https://routinator.do.nlnetlabs.nl/log

It all still works without any extra configuration in Routinator. 

Cheers,

Alex

> On 14 Jun 2023, at 15:15, Carlos Friaças via NANOG <nanog at nanog.org> wrote:
> 
> 
> Hi All,
> 
> Did this issue resurface some days ago...?
> I had nearly 6000 ROAs on June 1st.
> That went to ZERO on June 2nd.
> 
> I'm using routinator. Should i have changed something in my config to accomodate for some change?
> 
> Best Regards,
> Carlos
> 
> 
> 
> On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:
> 
>> Hi Job,
>> Thank you for this good analysis and for sharing your findings.
>> The issue has since been fixed and the team will publish a post-mortem accordingly once we are done with making sure the issue will not
>> reappear.
>> Your recommendation is well noted and I cc my colleague so that they can take that into consideration in our improvement roadmap.
>> Best regards,
>> ==============================
>> Cedrick Adrien MBEYET
>> Ebene Cybercity, Mauritius
>> +230 5851 7674
>> +++ Never give up, Keep moving forward +++
>> On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG <nanog at nanog.org> wrote:
>>      Hi all,
>> 
>>      It appears PacketVis correctly identified an issue.
>> 
>>      AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
>>      'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
>>      'K1eJenypZMPIt_e92qek2jSpj4A.mft'.
>> 
>>      The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
>>      Authorities. This Manifest represents the demarcation point between
>>      "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
>>      members". In other words; this is an important top-level Manifest in the
>>      critical path towards the ROAs of the Afrinic members.
>> 
>>      There was a ~ 7 hour gap in the validity window of this Manifest and its
>>      companion CRL (from 20221120T000311Z until 20221120T071514Z). The
>>      serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.
>> 
>>      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
>>          CRL Serial Number:        1E19
>>          CRL valid since:          Nov 18 00:03:11 2022 GMT
>>          CRL valid until:          Nov 20 00:03:11 2022 GMT
>> 
>>          CRL Serial Number:        1E1A
>>          CRL valid since:          Nov 20 07:15:12 2022 GMT
>>          CRL valid until:          Nov 22 07:15:12 2022 GMT
>> 
>>      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
>>          Manifest Number:          12B2
>>          Manifest valid since:     Nov 18 00:03:13 2022 GMT
>>          Manifest valid until:     Nov 20 00:03:13 2022 GMT
>> 
>>          Manifest Number:          12B3
>>          Manifest valid since:     Nov 20 07:15:14 2022 GMT
>>          Manifest valid until:     Nov 22 07:15:14 2022 GMT
>> 
>>      (The above can be reconstructed using archives from http://www.rpkiviews.org)
>> 
>>      The rcynic validator hosted at Afrinic also noticed a gap in objects:
>>      https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html
>> 
>>      A possible recommendation might be to increase the validity window of
>>      these two objects from a sliding 48-hour window to a 1 or 2 week window.
>>      This way any stalling in the issuance process wouldn't case operational
>>      issues on the weekend.
>> 
>>      Kind regards,
>> 
>>      Job
>> 
>>      [1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
>>      [2]: SKI 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80
>> 
>>      On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
>>      > From: PacketVis <notifications at packetvis.com>
>>      > Date: Sun, 20 Nov 2022 04:30:44 +0000
>>      >
>>      > Possible TA malfunction or incomplete VRP file: 73.95% of the ROAs disappeared from afrinic
>>      >
>>      > See more details about the event:
>>      >https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342fac
>>      a613
>> 