Europe-to-US congestion and packet loss on he.net network, and their NOC@ won't even respond

Sun Dec 1 09:11:54 UTC 2013

On 2013-W48-6 23:19 -0800, Matthew Petach wrote:
> On Fri, Nov 29, 2013 at 11:30 PM, Constantine A. Murenin <mureninc at gmail.com
> > wrote:
> 
> > Dear NANOG@,
> >
> 
> ...
> 
> 
> > From hetzner.de through he.net:
> >
> >
> > Cns# date ; mtr --report{,-wide,-cycles=600} --interval 0.1 --order "SRL
> > BGAWV" -4 ????c????????.indiana.edu ; date
> >
> 
> 
> Using a 1/10th of a second interval is rather anti-social.
> I know we rate-limit ICMP traffic down, and such a
> short interval would be detected as attack traffic,
> and treated as such.
> 
> I would take any results you get from such probes
> with a grain of salt.  What results do you get with
> a more sane interval, one of at least 1 second or
> more?
> 
> Matt

For what it is worth, I used to think the same, until I saw several 
providers themselves suggest that 1000 packets should be sent, with 
the 0.1 s interval.  So, this is considered normal and appropriate 
nowadays.

Anyhow, is this better?

I now saw a 2% traffic loss this night at a random test time, and 
the 151ms avg rtt on this 114ms rtt route.

Cns# date ; mtr --report{,-wide,-cycles=600} --interval 0.5 --order "SRL BGAWV" -4 ????c????????.indiana.edu ; date
Sat Nov 30 23:17:13 PST 2013
HOST: Cns???????                                   Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1.|-- static.??.???.4.46.clients.your-server.de    600   600  0.0%    0.5   1.0   1.3   4.6   1.1
  2.|-- hos-tr1.juniper1.rz13.hetzner.de             600   600  0.0%    0.1   0.2   2.0  58.5   7.9
  3.|-- core21.hetzner.de                            600   600  0.0%    0.2   0.2   0.2  10.2   0.7
  4.|-- core22.hetzner.de                            600   600  0.0%    0.2   0.2   0.2  11.2   0.8
  5.|-- core1.hetzner.de                             600   600  0.0%    4.8   4.8   4.8  25.1   1.3
  6.|-- juniper1.ffm.hetzner.de                      600   600  0.0%    4.8   4.8   4.8  13.9   0.6
  7.|-- 30gigabitethernet1-3.core1.ams1.he.net       600   595  0.8%   11.2  14.3  15.2 121.4   7.4
  8.|-- 10gigabitethernet1-4.core1.lon1.he.net       600   600  0.0%   18.2  21.0  21.3  51.2   4.0
  9.|-- 10gigabitethernet10-4.core1.nyc4.he.net      600   592  1.3%   86.9 125.9 126.4 160.7  10.6
 10.|-- 100gigabitethernet7-2.core1.chi1.he.net      600   591  1.5%  106.6 145.1 145.4 190.9  10.5
 11.|-- ???                                          600     0 100.0    0.0   0.0   0.0   0.0   0.0
 12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net   600   589  1.8%  114.3 148.9 149.2 167.9   9.1
 13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu         600   589  1.8%  113.4 149.2 149.5 173.4   9.3
 14.|-- ae-0.0.br2.bldc.net.uits.iu.edu              600   590  1.7%  114.5 150.2 150.5 175.6   9.3
 15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu             600   589  1.8%  114.3 150.5 150.8 181.0   9.1
 16.|-- ????c????????.indiana.edu                    600   589  1.8%  114.8 150.7 151.0 170.7   9.0
Sat Nov 30 23:24:06 PST 2013

The ICMP timestamp request/reply test still indicates that only 
one path is affected:  the one from Europe to US over he.net.

Cns# date ; unbuffer hping --icmp-ts --count 30 ????c????????.indiana.edu | \
perl -ne 'if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {($s, $p) = ($1, $2);} \
if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {($o, $r, $t) = ($1, $2, $3);} \
if (/tsrtt=(\d+)/) { \
print $s, "\t", $p, "\t", $1, " = ", $r - $o, " + ", $o + $1 - $t, "\n"; }'
Sun Dec  1 00:55:46 PST 2013
0       151.3   151 = 91 + 60
1       154.2   154 = 93 + 61
2       127.8   127 = 67 + 60
3       123.6   123 = 63 + 60
4       136.9   137 = 76 + 61
5       149.6   149 = 89 + 60
6       147.4   147 = 87 + 60
7       133.5   133 = 73 + 60
8       152.2   152 = 92 + 60
9       137.3   137 = 77 + 60
10      143.7   144 = 84 + 60
11      124.5   124 = 64 + 60
12      141.4   141 = 81 + 60
13      118.0   118 = 58 + 60
14      153.6   154 = 94 + 60
15      137.7   138 = 78 + 60
16      119.9   120 = 60 + 60
17      130.6   131 = 71 + 60
18      144.6   145 = 85 + 60
19      138.8   139 = 79 + 60
20      155.7   156 = 96 + 60
21      128.8   129 = 69 + 60
22      153.0   153 = 93 + 60
23      146.5   147 = 87 + 60
24      137.2   138 = 77 + 61
25      153.3   154 = 94 + 60
26      146.3   147 = 87 + 60
27      150.1   151 = 91 + 60
28      150.5   150 = 90 + 60
29      143.5   143 = 83 + 60

Cheers,
Constantine.