GPS Sync Outage

Wed Jan 1 01:52:02 UTC 2020

Greetings,

This email is a response to https://mailman.nanog.org/pipermail/nanog/2020-January/105058.html
in which Forrest discussed GPS time sync issues today.

The galmon.eu project also monitors GPS, and one thing we found is that at
20:39:47 UTC today, which is near 2PM MST, GPS PRN 29 emitted a clock update
that is pretty weird. 

The 't0c' is the t=0 of the clock correction parameters and it went
backwards by 16 seconds, and it introduced a 5.9 nanosecond clock jump.

This was recorded by 18 of our stations, mostly in Europe and the US east
coast: 

Tue, 31 Dec 2019 20:39:47.0000 +0000 src 33 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749  Timejump: -5.89898 after -16 seconds, old t0c 15750
Tue, 31 Dec 2019 20:39:47.0005 +0000 src 13 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9999 +0000 src 3 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9997 +0000 src 15 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9998 +0000 src 101 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9996 +0000 src 32 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9996 +0000 src 57 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.9999 +0000 src 37 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.0000 +0000 src 19 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:47.0004 +0000 src 49 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0000 +0000 src 27 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0005 +0000 src 56 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0004 +0000 src 41 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.4453 +0000 src 14 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0001 +0000 src 141 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0002 +0000 src 40 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0003 +0000 src 102 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749
Tue, 31 Dec 2019 20:39:48.0002 +0000 src 58 imptow 247203 GPS 29 at 0: 247206 frame 1 gpshealth = 0, wn 2086 t0c 15749

This was the previous t0c:

Tue, 31 Dec 2019 20:39:18.0000 +0000 src 27 imptow 247173 GPS 29 at 0: 247176 frame 1 gpshealth = 0, wn 2086 t0c 15750

And this is the next one:

Tue, 31 Dec 2019 21:59:46.9996 +0000 src 33 imptow 252003 GPS 29 at 0: 252006 frame 1 gpshealth = 0, wn 2086 t0c 16199  Timejump: -0.192813 after 7200 seconds, old t0c 15749

We spend more of our time looking at Galileo, but we have all the GPS data.
I haven't looked enough at GPS to see if "t0c going backwards" is a common
thing, but it does not look common.

Perhaps this could be related. If anyone wants a copy of GPS traffic from
today, let me know.

	Bert