afrinic rpki issue

Wed Jun 14 14:21:59 UTC 2023

On Wed, 14 Jun 2023, Alex Band wrote:

> Hi Carlos,

Hi Alex, All,

> Because of the issues that AfriNIC is facing, they are forcing all 
> traffic from HTTPS to rsync, so you should check if rsync can properly 
> set up outbound connections from your machine. What?s the output you get 
> when you rsync rsync://rpki.afrinic.net/repository/ ?

drwxr-xr-x          4,096 2023/06/14 12:04:28 .
-rw-r--r--            496 2020/04/08 19:58:03 AfriNIC-simple.tal
-rw-r--r--          1,216 2020/03/30 13:00:32 AfriNIC.cer
drwxr-xr-x          4,096 2023/06/09 13:50:13 
04E8B0D80F4D11E0B657D8931367AE7D
drwxr-xr-x         32,768 2023/06/14 12:04:28 afrinic
drwxr-xr-x          4,096 2023/06/14 01:05:30 apnic
drwxr-xr-x          8,192 2023/06/14 11:42:38 arin
drwxr-xr-x            120 2023/06/14 01:15:32 lacnic
drwxr-xr-x         16,384 2023/06/14 12:04:01 member_repository
drwxr-xr-x          4,096 2023/06/14 01:20:30 ripe

Seems to be working...

> I do an interactive Routinator validation run with debug logging enabled, like so:
>
> $ routinator -vv vrps -f summary
>
> Then I see the following in the logs:
>
> [WARN] RRDP https://rrdp.afrinic.net/notification.xml: Getting notification file failed with status 204 No Content
> [INFO] RRDP https://rrdp.afrinic.net/notification.xml: Update failed and current copy is expired since 2023-05-30 10:43:44 UTC.
> [INFO] RRDP https://rrdp.afrinic.net/notification.xml: Falling back to rsync.
> [INFO] rsyncing from rsync://rpki.afrinic.net/repository/.
>

Found valid trust anchor https://rpki.afrinic.net/repository/AfriNIC.cer. 
Processing.
RRDP https://rrdp.afrinic.net/notification.xml: Updating server
RRDP https://rrdp.afrinic.net/notification.xml: malformed XML
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/62gPOPXWxxu0sQa4vQZYUBLaMbY.mft: 
failed to validate
CA for 
rsync://rpki.afrinic.net/repository/04E8B0D80F4D11E0B657D8931367AE7D/ 
rejected, resources marked as unsafe:
    0.0.0.0/0
    ::/0
    AS0-AS4294967295

> Then, rsyncing the contents works just fine; objects are fetched and validated. Some objects fail validation with "certificate is not yet valid.?, "certificate has been revoked.? and ?Object not found.? but that appears unrelated to the connectivity issues they?re facing.
>
> I end up with the following totals:
>
> Summary at 2023-06-14 13:43:24.366013 UTC
> afrinic:              ROAs:    5756 verified;
>            VRPs:    7121 verified,    6820 final;
>    router certs:       0 verified;
>     router keys:       0 verified,       0 final.
>           ASPAs:       0 verified,       0 final.

Where do you see this?
Command output?

Summary at 2023-06-14 14:11:34.413948850 UTC
ripe: 39230 verified ROAs, 212122 verified VRPs, 6 unsafe VRPs, 212117 
final VRPs.
apnic: 24878 verified ROAs, 111967 verified VRPs, 0 unsafe VRPs, 111699 
final VRPs.
arin: 64077 verified ROAs, 79176 verified VRPs, 0 unsafe VRPs, 78064 final 
VRPs.
lacnic: 17966 verified ROAs, 32624 verified VRPs, 5 unsafe VRPs, 31033 
final VRPs.
afrinic: 0 verified ROAs, 0 verified VRPs, 0 unsafe VRPs, 0 final VRPs.
total: 146151 verified ROAs, 435889 verified VRPs, 11 unsafe VRPs, 432913 
final VRPs.

> If you want some logs to compare, you can have a look here:
> https://routinator.do.nlnetlabs.nl/log

Thanks.

> It all still works without any extra configuration in Routinator.

Well, for me it's still not really working yet.... :-)

Thanks anyway.

Cheers,
Carlos

> Cheers,
>
> Alex
>
>
>
>> On 14 Jun 2023, at 15:15, Carlos Friaças via NANOG <nanog at nanog.org> wrote:
>>
>>
>> Hi All,
>>
>> Did this issue resurface some days ago...?
>> I had nearly 6000 ROAs on June 1st.
>> That went to ZERO on June 2nd.
>>
>> I'm using routinator. Should i have changed something in my config to accomodate for some change?
>>
>> Best Regards,
>> Carlos
>>
>>
>>
>> On Sun, 20 Nov 2022, Cedrick Adrien Mbeyet wrote:
>>
>>> Hi Job,
>>> Thank you for this good analysis and for sharing your findings.
>>> The issue has since been fixed and the team will publish a post-mortem accordingly once we are done with making sure the issue will not
>>> reappear.
>>> Your recommendation is well noted and I cc my colleague so that they can take that into consideration in our improvement roadmap.
>>> Best regards,
>>> ==============================
>>> Cedrick Adrien MBEYET
>>> Ebene Cybercity, Mauritius
>>> +230 5851 7674
>>> +++ Never give up, Keep moving forward +++
>>> On Sun, Nov 20, 2022 at 3:49 PM Job Snijders via NANOG <nanog at nanog.org> wrote:
>>>      Hi all,
>>>
>>>      It appears PacketVis correctly identified an issue.
>>>
>>>      AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
>>>      'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
>>>      'K1eJenypZMPIt_e92qek2jSpj4A.mft'.
>>>
>>>      The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
>>>      Authorities. This Manifest represents the demarcation point between
>>>      "Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
>>>      members". In other words; this is an important top-level Manifest in the
>>>      critical path towards the ROAs of the Afrinic members.
>>>
>>>      There was a ~ 7 hour gap in the validity window of this Manifest and its
>>>      companion CRL (from 20221120T000311Z until 20221120T071514Z). The
>>>      serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.
>>>
>>>      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
>>>          CRL Serial Number:        1E19
>>>          CRL valid since:          Nov 18 00:03:11 2022 GMT
>>>          CRL valid until:          Nov 20 00:03:11 2022 GMT
>>>
>>>          CRL Serial Number:        1E1A
>>>          CRL valid since:          Nov 20 07:15:12 2022 GMT
>>>          CRL valid until:          Nov 22 07:15:12 2022 GMT
>>>
>>>      rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
>>>          Manifest Number:          12B2
>>>          Manifest valid since:     Nov 18 00:03:13 2022 GMT
>>>          Manifest valid until:     Nov 20 00:03:13 2022 GMT
>>>
>>>          Manifest Number:          12B3
>>>          Manifest valid since:     Nov 20 07:15:14 2022 GMT
>>>          Manifest valid until:     Nov 22 07:15:14 2022 GMT
>>>
>>>      (The above can be reconstructed using archives from http://www.rpkiviews.org)
>>>
>>>      The rcynic validator hosted at Afrinic also noticed a gap in objects:
>>>      https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html
>>>
>>>      A possible recommendation might be to increase the validity window of
>>>      these two objects from a sliding 48-hour window to a 1 or 2 week window.
>>>      This way any stalling in the issuance process wouldn't case operational
>>>      issues on the weekend.
>>>
>>>      Kind regards,
>>>
>>>      Job
>>>
>>>      [1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
>>>      [2]: SKI 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80
>>>
>>>      On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
>>>     > From: PacketVis <notifications at packetvis.com>
>>>     > Date: Sun, 20 Nov 2022 04:30:44 +0000
>>>     >
>>>     > Possible TA malfunction or incomplete VRP file: 73.95% of the ROAs disappeared from afrinic
>>>     >
>>>     > See more details about the event:
>>>     >https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342fac
>>>      a613
>>>
>
>