<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On 11 Jan 2019, at 3:59 PM, John Curran <<a href="mailto:jcurran@arin.net" class="">jcurran@arin.net</a>> wrote:<br class="">
<div>
<blockquote type="cite" class="">...<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">My
apologies for this incident – while ARIN does have some fragility in our older systems (which we have been working aggressively to phase out via system refresh and replacements), it is not acceptable to have this situation with key infrastructure such as our
DNS zones. We will prioritize the necessary alert and monitor changes and I will report back to the community once that has been completed.</span></div>
</blockquote>
</div>
<div class=""><br class="">
</div>
<div class="">Folks - </div>
<div class=""><br class="">
</div>
<div class="">I indicated that we would report back once appropriate DNSSEC monitoring is in place - this has now been completed (ref: attached announcement of same)</div>
<div class=""><br class="">
</div>
<div class="">Thanks again for your patience in this matter,</div>
<div class="">/John</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">John Curran</div>
<div class="">President and CEO</div>
<div class="">American Registry for Internet Numbers</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Begin forwarded message:</div>
<br class="Apple-interchange-newline">
<div style="margin: 0px;" class=""></div>
</div>
<blockquote type="cite" class="">
<div class="">
<div style="margin: 0px;" class=""><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><b class="">From: </b></span><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class="">ARIN
<<a href="mailto:info@arin.net" class="">info@arin.net</a>><br class="">
</span></div>
<div style="margin: 0px;" class=""><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><b class="">Subject: </b></span><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><b class="">[arin-announce]
DNSSEC Monitoring Enhancements</b><br class="">
</span></div>
<div style="margin: 0px;" class=""><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><b class="">Date: </b></span><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class="">4
February 2019 at 11:32:25 AM EST<br class="">
</span></div>
<div style="margin: 0px;" class=""><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><b class="">To: </b></span><span style="font-family: -webkit-system-font, "Helvetica Neue", Helvetica, sans-serif;" class=""><<a href="mailto:arin-announce@arin.net" class="">arin-announce@arin.net</a>><br class="">
</span></div>
<br class="">
<div class="">On 31 January, ARIN deployed DNSSEC monitoring enhancements, including proactive RRSIG expiration checking, zone syntax checking, and DNSSEC validation. We are monitoring from various disparate locations across the Internet with these checks.
This effort was undertaken in response to the incident that occurred on 11 January, detailed in the incident report below.<br class="">
<br class="">
Improved monitoring of DNSSEC and the <a href="http://arin.net" class="">arin.net</a> zone will provide earlier alerts of any issues such as Resource Record Signature (RRSIG) expiration and any issues with DNSSEC validation. These enhancements will provide
early warning of potential issues, prevent outages, and improve our ability to troubleshoot DNSSEC problems if they occur in the future.<br class="">
<br class="">
Regards,<br class="">
Mark Kosters<br class="">
Chief Technology Officer<br class="">
American Registry for Internet Numbers (ARIN)<br class="">
<br class="">
Incident Report:<br class="">
<br class="">
On 11 January 2019, at approximately 8:30 a.m. ET, ARIN monitoring systems alerted that some
<a href="http://arin.net" class="">arin.net</a> properties were unreachable. All users with validating DNS resolvers were unable to look up resources within
<a href="http://arin.net" class="">arin.net</a> and thus were unable to reach them. ARIN’s
<a href="http://www.arin.net" class="">www.arin.net</a> and <a href="http://ftp.arin.net" class="">
ftp.arin.net</a> sites and Whois, RPKI, and DNS services were affected for those users who use validating resolvers.<br class="">
<br class="">
ARIN’s Engineering staff determined that DNSSEC validation for the <a href="http://arin.net" class="">
arin.net</a> zone was failing and temporarily unpublished Delegation Signer (DS) records with our registrar so that we could investigate the problem. Upon troubleshooting, ARIN staff discovered that the removal of a resource record had created a spurious record,
which caused a script to fail to reload. New versions of the zone could not be loaded, and the zone file in use expired. After determining the cause of the problem, the offending file was removed and the zone was reloaded. Delegation Signer (DS) records were
republished and the zone validated, restoring service at approximately 10:30 a.m. ET.<br class="">
<br class="">
_______________________________________________<br class="">
ARIN-Announce</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
</body>
</html>