Recent NTP pool traffic increase (update)

Thu Dec 22 03:13:42 UTC 2016

Hello.

I operate the public NTP Service as 133.100.9.2
and 133.100.11.8 at Fukuoka University, Japan.
I have a lot of trouble with too much NTP traffic from
many routers which 133.100.9.2 as default setting of NTP
has been set like Tenda or LB-Link etc.
So, although I'd like to contact Firmware developpers of these company
and would like them to change the default settins,
is there the person knowing the contact information?

-- 
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan

2016-12-21 18:36 GMT+09:00 Denys Fedoryshchenko <denys at visp.net.lb>:
> Hello,
>
> I'm not sure i should continue to CC nanog, if someone interested to be in
> CC for further updates this story please let me know.
>
> TP-Link not related, it was misunderstanding or wrong customer report.
>
> Tenda routers i believe most of cheap models are affected by this problem.
> On ISPs i have access i see too many of them sending requests to 133.100.9.2
> (if it is unreachable, repeating each 10 seconds), this particular ip seems
> hardcoded there. I am sure as soon as your server is down, way they coded -
> it will make all this routers to ddos this particular ip, repeating NTP
> queries very often without any backoff/protection mechanism.
> Particular model i tested - W308R / Wireless N300 Home Router, but i believe
> many models are affected.
> Firmware: System Version: 5.0.7.53_en hw version : v1.0
>
> Another possible vendor is LB-Link, but i dont have yet any info from
> customers who own them.
>
> On 2016-12-21 11:00, FUJIMURA Sho wrote:
>>
>> Hello.
>>
>> I'm Sho FUJIMURA.
>> Thank you for your information.
>> I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
>> I'd like to reduce the traffic because I have trouble with too much
>> traffic recently.
>> So, I'm interested in the root of the the problem.
>> If possible, would you please tell me the model numbers of Tenda and
>> TP-Link??
>>
>> --
>> Sho FUJIMURA
>> Information Technology Center, Fukuoka University.
>> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
>>
>> fujimura at fukuoka-u.ac.jp
>>
>> 2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys at visp.net.lb>:
>>
>>> I'm not sure if this issue relevant to discussed topic, Tenda
>>> routers here for a while on market, and i think i noticed this issue
>>> just now,
>>> because NTP servers they are using supposedly for healthcheck went
>>> down (or NTP owners blocked ISP's i support, due such routers).
>>>
>>> At least after checking numerous users, i believe Tenda hardcoded
>>> those NTP IPs. What worsen issue, that in Lebanon several times per
>>> day, for example at 18pm - short electricity cutoff,
>>> and majority of users routers will reboot and surely reconnect, so
>>> it will look like a countrywide spike in NTP traffic.
>>>
>>> I checked for a 10min also this NTP ips in dns responses, none of
>>> thousands of users tried to resolve any name with them over any DNS
>>> server, so i conclude they are hardcoded somewhere in firmware.
>>>
>>> Here is traffic of Tenda router after reconnecting (but not full
>>> powercycle, i dont have it in my hands). But as you can see, no DNS
>>> resolution attempts:
>>>
>>> 20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
>>> S=XXXXXX M=Authentication succeeded
>>> 20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>>> length 12
>>> 20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>>> length 24
>>> 20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
>>> length 12
>>> 20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
>>> length 18
>>> 20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
>>> length 24
>>> 20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
>>> length 24
>>> 20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 133.100.9.2.123: NTPv3, Client, length 48
>>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 192.5.41.41.123: NTPv3, Client, length 48
>>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>>> 192.5.41.40.123: NTPv3, Client, length 48
>>>
>>> Here is example of Tenda traffic if it is unable to reach
>>> destination, it repeats request each 10 seconds endlessly, my guess
>>> they are using ntp to show
>>> status of internet connection.
>>> So, now that NTP servers getting quite significant DDoS such way.
>>>
>>> 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>>> 22:57:43)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177063.000000000
>>> (2016/12/19 22:57:43)
>>> 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>>> 22:57:43)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177063.000000000
>>> (2016/12/19 22:57:43)
>>> 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>>> 22:57:43)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177063.000000000
>>> (2016/12/19 22:57:43)
>>> 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>>> 22:57:53)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177073.000000000
>>> (2016/12/19 22:57:53)
>>> 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>>> 22:57:53)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177073.000000000
>>> (2016/12/19 22:57:53)
>>> 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags
>>> [none], proto UDP (17), length 76)
>>> 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
>>> 48
>>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>>> 0 (1s), precision 0
>>> Root Delay: 0.000000, Root dispersion: 0.000000,
>>> Reference-ID: (unspec)
>>> Reference Timestamp:  0.000000000
>>> Originator Timestamp: 0.000000000
>>> Receive Timestamp:    0.000000000
>>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>>> 22:57:53)
>>> Originator - Receive Timestamp:  0.000000000
>>> Originator - Transmit Timestamp: 3691177073.000000000
>>> (2016/12/19 22:57:53)
>>>
>>> On 2016-12-19 21:40, Roland Dobbins wrote:
>>> On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
>>>
>>> If it is necessary i can investigate further.
>>>
>>> Yes, please!
>>>
>>> -----------------------------------
>>> Roland Dobbins <rdobbins at arbor.net>
>
>