Europe-to-US congestion and packet loss on he.net network, and their NOC@ won't even respond

Constantine A. Murenin mureninc at gmail.com
Wed Dec 4 08:34:07 UTC 2013


For anyone's following the story:  the weeks-long congestion on he.net remains, 
however, hetzner has switched my original route to an alternative uplink.  
I'm no longer experiencing the he.net evening jitter that would bring my 
avg rtt from the non-congested 114ms to an average of 140ms and more 
during the busy times.

Other sites that have an uplink from he.net are still affected, 
with a noticeable jitter, delay and some packet loss (which 
seems to increase with the increase in the avg rtt).


Was:

Tue Dec  3 22:25:38 PST 2013
Wed Dec  4 07:25:38 CET 2013
HOST: Cns???????                                   Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1.|-- static.??.???.4.46.clients.your-server.de    900   900  0.0%    0.5   0.9   1.3   7.3   1.2
  2.|-- hos-tr1.juniper1.rz13.hetzner.de             900   900  0.0%    0.2   0.2   2.1 101.2   9.0
  3.|-- core21.hetzner.de                            900   900  0.0%    0.2   0.2   0.2  19.7   1.1
  4.|-- core22.hetzner.de                            900   897  0.3%    0.2   0.2   0.2  19.6   1.1
  5.|-- core1.hetzner.de                             900   900  0.0%    4.8   4.8   4.8  16.4   1.1
  6.|-- juniper1.ffm.hetzner.de                      900   893  0.8%    4.8   4.8   4.8  19.1   0.7
  7.|-- 30gigabitethernet1-3.core1.ams1.he.net       900   900  0.0%   11.2  14.0  14.8 123.4   6.5
  8.|-- 10ge1-4.core1.lon1.he.net                    900   900  0.0%   18.2  19.9  20.2  59.7   3.9
  9.|-- 10ge10-4.core1.nyc4.he.net                   900   896  0.4%   86.9 113.6 114.2 148.4  11.4
 10.|-- 100ge7-2.core1.chi1.he.net                   900   898  0.2%  106.6 133.0 133.6 184.6  12.4
 11.|-- ???                                          900     0 100.0    0.0   0.0   0.0   0.0   0.0
 12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net   900   899  0.1%  113.3 136.7 137.1 162.3  10.8
 13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu         900   895  0.6%  113.3 137.2 137.7 183.3  11.1
 14.|-- ae-0.0.br2.bldc.net.uits.iu.edu              900   900  0.0%  114.3 137.8 138.2 162.8  10.7
 15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu             900   899  0.1%  114.4 137.9 138.4 167.5  10.7
 16.|-- m???c????????.indiana.edu                    900   898  0.2%  114.5 138.1 138.5 159.0  10.4
Tue Dec  3 22:43:16 PST 2013
Wed Dec  4 07:43:16 CET 2013
0       154.8   155 = 94 + 61
1       127.9   128 = 67 + 61
2       125.0   125 = 64 + 61
3       131.5   132 = 71 + 61
4       131.2   132 = 71 + 61
5       117.4   118 = 57 + 61
6       132.9   133 = 72 + 61
7       143.4   144 = 83 + 61
8       138.7   139 = 78 + 61
9       131.5   131 = 70 + 61
10      142.2   142 = 81 + 61
11      152.2   152 = 91 + 61
12      125.2   125 = 64 + 61
13      125.6   125 = 64 + 61
14      140.8   141 = 80 + 61
15      150.9   151 = 89 + 62
Tue Dec  3 22:43:31 PST 2013
Wed Dec  4 07:43:31 CET 2013


Now:

Tue Dec  3 23:01:20 PST 2013
Wed Dec  4 08:01:20 CET 2013
HOST: Cns???????                                          Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1.|-- static.??.???.4.46.clients.your-server.de           900   900  0.0%    0.5   0.9   1.3   7.0   1.1
  2.|-- hos-tr1.juniper1.rz13.hetzner.de                    900   900  0.0%    0.1   0.2   4.3 245.4  18.9
  3.|-- core21.hetzner.de                                   900   900  0.0%    0.2   0.2   0.3  99.4   3.5
  4.|-- core12.hetzner.de                                   900   900  0.0%    2.7   2.8   2.8  23.1   1.9
  5.|-- juniper4.rz2.hetzner.de                             900   899  0.1%    2.8   2.8   2.8  29.0   1.1
  6.|-- te0-0-2-1.nr21.b040138-0.nue01.atlas.cogentco.com   900   900  0.0%    3.1   3.3   3.4   7.1   0.6
  7.|-- te2-4.ccr01.nue01.atlas.cogentco.com                900   900  0.0%    3.1   6.6  25.9 244.4  51.0
    |  `|-- 154.25.0.13
  8.|-- te0-2-0-3.ccr22.muc01.atlas.cogentco.com            900   900  0.0%    5.7   5.8   5.9  11.1   0.5
  9.|-- be2229.ccr22.fra03.atlas.cogentco.com               900   900  0.0%   11.1  11.4  11.4  14.4   0.5
    |  `|-- 154.54.38.189
 10.|-- te0-3-0-2.mpd22.ams03.atlas.cogentco.com            900   900  0.0%   17.7  17.9  17.9  49.1   1.1
    |  `|-- 154.54.39.193
 11.|-- be2276.ccr22.lon13.atlas.cogentco.com               900   900  0.0%   25.3  27.4  27.4  33.2   1.3
    |  `|-- 154.54.37.106
    |   |-- 154.54.58.69
 12.|-- te0-7-0-33.ccr22.bos01.atlas.cogentco.com           900   900  0.0%   93.1  93.9  93.9  98.8   1.2
    |  `|-- 154.54.44.189
    |   |-- 130.117.0.185
 13.|-- te7-8.ccr02.alb02.atlas.cogentco.com                900   900  0.0%   96.8 119.9 128.8 364.2  56.9
    |  `|-- 154.54.27.153
 14.|-- te7-8.ccr01.buf02.atlas.cogentco.com                900   900  0.0%  103.3 126.8 135.8 444.9  59.0
 15.|-- te0-5-0-2.ccr21.cle04.atlas.cogentco.com            900   900  0.0%  108.6 109.5 109.5 113.5   1.3
 16.|-- te3-2.ccr01.cmh02.atlas.cogentco.com                900   900  0.0%  111.1 134.8 143.5 467.0  59.4
 17.|-- te4-7.ccr01.cvg02.atlas.cogentco.com                900   900  0.0%  113.8 122.5 126.2 467.9  39.6
 18.|-- te3-3.ccr01.ind01.atlas.cogentco.com                900   899  0.1%  116.1 137.8 145.4 457.7  56.0
 19.|-- 38.104.214.6                                        900   900  0.0%  115.8 118.3 118.5 206.0   8.1
 20.|-- xe-0-1-0.11.rtr.ictc.indiana.gigapop.net            900   900  0.0%  115.9 116.7 116.7 128.4   1.5
 21.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu                900   900  0.0%  115.9 117.0 117.1 168.6   3.6
 22.|-- ae-0.0.br2.bldc.net.uits.iu.edu                     900   900  0.0%  116.8 117.7 117.7 130.8   1.8
 23.|-- ae-10.0.cr3.bldc.net.uits.iu.edu                    900   899  0.1%  116.8 117.8 117.8 135.1   1.9
 24.|-- m???c????????.indiana.edu                           900   899  0.1%  117.1 117.9 117.9 122.0   1.2
Tue Dec  3 23:19:56 PST 2013
Wed Dec  4 08:19:56 CET 2013
0       120.1   120 = 57 + 63
1       117.3   117 = 54 + 63
2       120.3   120 = 57 + 63
3       117.3   117 = 54 + 63
4       117.4   118 = 54 + 64
5       117.3   118 = 54 + 64
6       120.0   120 = 57 + 63
7       117.3   118 = 54 + 64
8       117.3   118 = 54 + 64
9       117.6   118 = 54 + 64
10      117.3   118 = 54 + 64
11      117.4   118 = 54 + 64
12      117.4   117 = 53 + 64
13      118.6   118 = 55 + 63
14      117.5   117 = 53 + 64
15      117.3   117 = 53 + 64
Tue Dec  3 23:20:11 PST 2013
Wed Dec  4 08:20:11 CET 2013


Yet:

Cns# sh -c 'while (true); do date; env TZ=Europe/Berlin date; mtr --report{,-wide,-cycles=900} --interval 1 --order "SRL BGAWV" -4 ntp1.yycix.ca; date; env TZ=Europe/Berlin date; unbuffer hping --icmp-ts --count 16 ntp1.yycix.ca | perl -ne "if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {(\$s,\$p)=(\$1,\$2);} if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {(\$o,\$r,\$t)=(\$1,\$2,\$3);} if (/tsrtt=(\d+)/) {print \$s,qq/\t/,\$p,qq/\t/,\$1,qq/ = /,\$r-\$o,qq/ + /,\$o+\$1-\$t,qq/\n/;}"; done'
Tue Dec  3 23:32:37 PST 2013
Wed Dec  4 08:32:37 CET 2013
HOST: Cns???????                                               Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1.|-- static.??.???.4.46.clients.your-server.de                900   900  0.0%    0.5   0.9   1.2   9.7   1.2
  2.|-- hos-tr4.juniper2.rz13.hetzner.de                         900   900  0.0%    0.2   0.2   2.0 138.3   8.9
  3.|-- core22.hetzner.de                                        900   900  0.0%    0.2   0.2   0.2   8.5   0.4
  4.|-- core1.hetzner.de                                         900   900  0.0%    4.8   4.8   4.8  16.0   0.7
  5.|-- juniper1.ffm.hetzner.de                                  900   900  0.0%    4.8   4.8   4.8  13.8   0.7
  6.|-- 30gigabitethernet1-3.core1.ams1.he.net                   900   900  0.0%   11.2  14.2  14.9 106.2   6.2
  7.|-- 10ge1-4.core1.lon1.he.net                                900   900  0.0%   18.0  20.5  20.8  55.6   4.1
  8.|-- 10ge10-4.core1.nyc4.he.net                               900   895  0.6%   86.4 119.3 120.0 158.6  13.1
  9.|-- 10ge1-2.core1.tor1.he.net                                900   896  0.4%   96.4 128.4 129.1 159.6  12.9
 10.|-- 10ge3-1.core1.ywg1.he.net                                900   895  0.6%  121.8 152.2 152.7 181.5  12.6
 11.|-- 10ge1-1.core1.yyc1.he.net                                900   888  1.3%  135.7 168.0 168.5 197.4  13.0
 12.|-- sebo-systems-inc.gigabitethernet2-23.core1.yyc1.he.net   900   890  1.1%  138.4 168.2 168.6 190.9  12.5
 13.|-- ???                                                      900     0 100.0    0.0   0.0   0.0   0.0   0.0
 14.|-- ntp1.yycix.ca                                            900   893  0.8%  138.7 168.8 169.2 191.9  12.2
Tue Dec  3 23:49:07 PST 2013
Wed Dec  4 08:49:07 CET 2013
0       180.0   180 = 109 + 71
1       155.9   156 = 84 + 72
2       143.6   144 = 72 + 72
3       179.8   180 = 108 + 72
4       161.0   161 = 89 + 72
5       155.9   156 = 84 + 72
6       168.0   168 = 97 + 71
7       164.5   165 = 93 + 72
8       177.9   178 = 107 + 71
9       159.6   160 = 88 + 72
10      179.1   180 = 108 + 72
11      178.6   178 = 106 + 72
12      149.8   149 = 78 + 71
13      166.2   166 = 94 + 72
14      146.8   147 = 75 + 72
15      153.7   154 = 82 + 72
Tue Dec  3 23:49:22 PST 2013
Wed Dec  4 08:49:22 CET 2013


Cheers,
Constantine.

On 2013-W48-7 01:11 -0800, Constantine A. Murenin wrote:
> Anyhow, is this better?
> 
> I now saw a 2% traffic loss this night at a random test time, and 
> the 151ms avg rtt on this 114ms rtt route.
> 
> 
> Cns# date ; mtr --report{,-wide,-cycles=600} --interval 0.5 --order "SRL BGAWV" -4 ????c????????.indiana.edu ; date
> Sat Nov 30 23:17:13 PST 2013
> HOST: Cns???????                                   Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
>   1.|-- static.??.???.4.46.clients.your-server.de    600   600  0.0%    0.5   1.0   1.3   4.6   1.1
>   2.|-- hos-tr1.juniper1.rz13.hetzner.de             600   600  0.0%    0.1   0.2   2.0  58.5   7.9
>   3.|-- core21.hetzner.de                            600   600  0.0%    0.2   0.2   0.2  10.2   0.7
>   4.|-- core22.hetzner.de                            600   600  0.0%    0.2   0.2   0.2  11.2   0.8
>   5.|-- core1.hetzner.de                             600   600  0.0%    4.8   4.8   4.8  25.1   1.3
>   6.|-- juniper1.ffm.hetzner.de                      600   600  0.0%    4.8   4.8   4.8  13.9   0.6
>   7.|-- 30gigabitethernet1-3.core1.ams1.he.net       600   595  0.8%   11.2  14.3  15.2 121.4   7.4
>   8.|-- 10gigabitethernet1-4.core1.lon1.he.net       600   600  0.0%   18.2  21.0  21.3  51.2   4.0
>   9.|-- 10gigabitethernet10-4.core1.nyc4.he.net      600   592  1.3%   86.9 125.9 126.4 160.7  10.6
>  10.|-- 100gigabitethernet7-2.core1.chi1.he.net      600   591  1.5%  106.6 145.1 145.4 190.9  10.5
>  11.|-- ???                                          600     0 100.0    0.0   0.0   0.0   0.0   0.0
>  12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net   600   589  1.8%  114.3 148.9 149.2 167.9   9.1
>  13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu         600   589  1.8%  113.4 149.2 149.5 173.4   9.3
>  14.|-- ae-0.0.br2.bldc.net.uits.iu.edu              600   590  1.7%  114.5 150.2 150.5 175.6   9.3
>  15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu             600   589  1.8%  114.3 150.5 150.8 181.0   9.1
>  16.|-- ????c????????.indiana.edu                    600   589  1.8%  114.8 150.7 151.0 170.7   9.0
> Sat Nov 30 23:24:06 PST 2013
> 
> 
> 
> The ICMP timestamp request/reply test still indicates that only 
> one path is affected:  the one from Europe to US over he.net.
> 
> 
> Cns# date ; unbuffer hping --icmp-ts --count 30 ????c????????.indiana.edu | \
> perl -ne 'if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {($s, $p) = ($1, $2);} \
> if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {($o, $r, $t) = ($1, $2, $3);} \
> if (/tsrtt=(\d+)/) { \
> print $s, "\t", $p, "\t", $1, " = ", $r - $o, " + ", $o + $1 - $t, "\n"; }'
> Sun Dec  1 00:55:46 PST 2013
> 0       151.3   151 = 91 + 60
> 1       154.2   154 = 93 + 61
> 2       127.8   127 = 67 + 60
> 3       123.6   123 = 63 + 60
> 4       136.9   137 = 76 + 61
> 5       149.6   149 = 89 + 60
> 6       147.4   147 = 87 + 60
> 7       133.5   133 = 73 + 60
> 8       152.2   152 = 92 + 60
> 9       137.3   137 = 77 + 60
> 10      143.7   144 = 84 + 60
> 11      124.5   124 = 64 + 60
> 12      141.4   141 = 81 + 60
> 13      118.0   118 = 58 + 60
> 14      153.6   154 = 94 + 60
> 15      137.7   138 = 78 + 60
> 16      119.9   120 = 60 + 60
> 17      130.6   131 = 71 + 60
> 18      144.6   145 = 85 + 60
> 19      138.8   139 = 79 + 60
> 20      155.7   156 = 96 + 60
> 21      128.8   129 = 69 + 60
> 22      153.0   153 = 93 + 60
> 23      146.5   147 = 87 + 60
> 24      137.2   138 = 77 + 61
> 25      153.3   154 = 94 + 60
> 26      146.3   147 = 87 + 60
> 27      150.1   151 = 91 + 60
> 28      150.5   150 = 90 + 60
> 29      143.5   143 = 83 + 60
> 
> 
> 
> Cheers,
> Constantine.




More information about the NANOG mailing list