Erroneous Leap Second Introduced at 2014-06-30 23:59:59 UTC

Tue Jul 1 19:20:12 UTC 2014

On Mon, Jun 30, 2014 at 7:27 PM, Majdi S. Abbas <msa at latt.net> wrote:
> On Mon, Jun 30, 2014 at 05:33:52PM -0700, Tim Heckman wrote:
>> I just was alerted to one of the systems I managed having a time skew
>> greater than 100ms from NTP sources. Upon further investigation it
>> seemed that the time was off by almost exactly 1 second.
>>
>> Looking back over our NTP monitoring, it would appear that this system
>> had a large time adjust at approximately 00:00 UTC:
>
>         Okay.  Do you have any logging configured (peerstats, etc?) for
> ntpd?

Our systems all have loopstats and peerstats logging enabled. I have
those log files available if interested. However, when I searched over
the files I wasn't able to find anything that seemed to indicate this
was the peer who told the system to introduce a leap second. That
said, I might just not know what to look for in the logs.

>> A few of our systems did alert early this morning, indicating they
>> were going to be receiving a leap second today. However, I was unable
>> to determine the exact cause for NTP believing a leap second should be
>> added. And after some time a few of the systems were no longer
>> indicating that a leap second would be introduced.
>
>         This can happen if a server is either passing along a leap
> notification that it received, or is configured to use a leapseconds
> file that is incorrect.

Correct, I was hoping to determine which peer it was so I can reach
out to them to make sure this doesn't bleed in to the pool at the end
of the year. I was also more-or-less curious how wide-spread of an
issue this was, but I'm starting to think I may have been the only
person to catch it in the act. :)

>> This specific system is hosted in AWS US-WEST-2C and uses the
>> 0.amazon.pool.ntp.org pool.
>
>         0 is just one server in the pool (whichever you draw by
> rotation); is this the only server you have configured?

We use 0.amazon.pool.ntp.org, 1.amazon.pool.ntp.org, and
2.amazon.pool.ntp.org. As with the other widely-used pool hostnames,
each of these is a round-robin DNS entry with 4 hosts and a TTL of
150s.

>         --msa

Thank you for getting back to me.

Cheers!
-Tim