Recent NTP pool traffic increase (update)

Wed Dec 21 09:36:49 UTC 2016

Hello,

I'm not sure i should continue to CC nanog, if someone interested to be 
in CC for further updates this story please let me know.

TP-Link not related, it was misunderstanding or wrong customer report.

Tenda routers i believe most of cheap models are affected by this 
problem.
On ISPs i have access i see too many of them sending requests to 
133.100.9.2 (if it is unreachable, repeating each 10 seconds), this 
particular ip seems hardcoded there. I am sure as soon as your server is 
down, way they coded - it will make all this routers to ddos this 
particular ip, repeating NTP queries very often without any 
backoff/protection mechanism.
Particular model i tested - W308R / Wireless N300 Home Router, but i 
believe many models are affected.
Firmware: System Version: 5.0.7.53_en hw version : v1.0

Another possible vendor is LB-Link, but i dont have yet any info from 
customers who own them.

On 2016-12-21 11:00, FUJIMURA Sho wrote:
> Hello.
> 
> I'm Sho FUJIMURA.
> Thank you for your information.
> I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
> I'd like to reduce the traffic because I have trouble with too much
> traffic recently.
> So, I'm interested in the root of the the problem.
> If possible, would you please tell me the model numbers of Tenda and
> TP-Link??
> 
> --
> Sho FUJIMURA
> Information Technology Center, Fukuoka University.
> 8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan
> 
> fujimura at fukuoka-u.ac.jp
> 
> 2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <denys at visp.net.lb>:
> 
>> I'm not sure if this issue relevant to discussed topic, Tenda
>> routers here for a while on market, and i think i noticed this issue
>> just now,
>> because NTP servers they are using supposedly for healthcheck went
>> down (or NTP owners blocked ISP's i support, due such routers).
>> 
>> At least after checking numerous users, i believe Tenda hardcoded
>> those NTP IPs. What worsen issue, that in Lebanon several times per
>> day, for example at 18pm - short electricity cutoff,
>> and majority of users routers will reboot and surely reconnect, so
>> it will look like a countrywide spike in NTP traffic.
>> 
>> I checked for a 10min also this NTP ips in dns responses, none of
>> thousands of users tried to resolve any name with them over any DNS
>> server, so i conclude they are hardcoded somewhere in firmware.
>> 
>> Here is traffic of Tenda router after reconnecting (but not full
>> powercycle, i dont have it in my hands). But as you can see, no DNS
>> resolution attempts:
>> 
>> 20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
>> S=XXXXXX M=Authentication succeeded
>> 20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>> length 12
>> 20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
>> length 24
>> 20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
>> length 12
>> 20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
>> length 18
>> 20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
>> length 24
>> 20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
>> length 24
>> 20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>> 133.100.9.2.123: NTPv3, Client, length 48
>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>> 192.5.41.41.123: NTPv3, Client, length 48
>> 20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
>> 192.5.41.40.123: NTPv3, Client, length 48
>> 
>> Here is example of Tenda traffic if it is unable to reach
>> destination, it repeats request each 10 seconds endlessly, my guess
>> they are using ntp to show
>> status of internet connection.
>> So, now that NTP servers getting quite significant DDoS such way.
>> 
>> 19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>> 22:57:43)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177063.000000000
>> (2016/12/19 22:57:43)
>> 19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>> 22:57:43)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177063.000000000
>> (2016/12/19 22:57:43)
>> 19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177063.000000000 (2016/12/19
>> 22:57:43)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177063.000000000
>> (2016/12/19 22:57:43)
>> 19:58:02.164781 IP (tos 0x0, ttl 64, id 38518, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>> 22:57:53)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177073.000000000
>> (2016/12/19 22:57:53)
>> 19:58:02.164884 IP (tos 0x0, ttl 64, id 38519, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>> 22:57:53)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177073.000000000
>> (2016/12/19 22:57:53)
>> 19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags
>> [none], proto UDP (17), length 76)
>> 172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
>> 48
>> Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
>> 0 (1s), precision 0
>> Root Delay: 0.000000, Root dispersion: 0.000000,
>> Reference-ID: (unspec)
>> Reference Timestamp:  0.000000000
>> Originator Timestamp: 0.000000000
>> Receive Timestamp:    0.000000000
>> Transmit Timestamp:   3691177073.000000000 (2016/12/19
>> 22:57:53)
>> Originator - Receive Timestamp:  0.000000000
>> Originator - Transmit Timestamp: 3691177073.000000000
>> (2016/12/19 22:57:53)
>> 
>> On 2016-12-19 21:40, Roland Dobbins wrote:
>> On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:
>> 
>> If it is necessary i can investigate further.
>> 
>> Yes, please!
>> 
>> -----------------------------------
>> Roland Dobbins <rdobbins at arbor.net>