Europe-to-US congestion and packet loss on he.net network, and their NOC@ won't even respond
Constantine A. Murenin
mureninc at gmail.com
Wed Dec 4 08:34:07 UTC 2013
For anyone's following the story: the weeks-long congestion on he.net remains,
however, hetzner has switched my original route to an alternative uplink.
I'm no longer experiencing the he.net evening jitter that would bring my
avg rtt from the non-congested 114ms to an average of 140ms and more
during the busy times.
Other sites that have an uplink from he.net are still affected,
with a noticeable jitter, delay and some packet loss (which
seems to increase with the increase in the avg rtt).
Was:
Tue Dec 3 22:25:38 PST 2013
Wed Dec 4 07:25:38 CET 2013
HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev
1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.3 7.3 1.2
2.|-- hos-tr1.juniper1.rz13.hetzner.de 900 900 0.0% 0.2 0.2 2.1 101.2 9.0
3.|-- core21.hetzner.de 900 900 0.0% 0.2 0.2 0.2 19.7 1.1
4.|-- core22.hetzner.de 900 897 0.3% 0.2 0.2 0.2 19.6 1.1
5.|-- core1.hetzner.de 900 900 0.0% 4.8 4.8 4.8 16.4 1.1
6.|-- juniper1.ffm.hetzner.de 900 893 0.8% 4.8 4.8 4.8 19.1 0.7
7.|-- 30gigabitethernet1-3.core1.ams1.he.net 900 900 0.0% 11.2 14.0 14.8 123.4 6.5
8.|-- 10ge1-4.core1.lon1.he.net 900 900 0.0% 18.2 19.9 20.2 59.7 3.9
9.|-- 10ge10-4.core1.nyc4.he.net 900 896 0.4% 86.9 113.6 114.2 148.4 11.4
10.|-- 100ge7-2.core1.chi1.he.net 900 898 0.2% 106.6 133.0 133.6 184.6 12.4
11.|-- ??? 900 0 100.0 0.0 0.0 0.0 0.0 0.0
12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net 900 899 0.1% 113.3 136.7 137.1 162.3 10.8
13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 900 895 0.6% 113.3 137.2 137.7 183.3 11.1
14.|-- ae-0.0.br2.bldc.net.uits.iu.edu 900 900 0.0% 114.3 137.8 138.2 162.8 10.7
15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 900 899 0.1% 114.4 137.9 138.4 167.5 10.7
16.|-- m???c????????.indiana.edu 900 898 0.2% 114.5 138.1 138.5 159.0 10.4
Tue Dec 3 22:43:16 PST 2013
Wed Dec 4 07:43:16 CET 2013
0 154.8 155 = 94 + 61
1 127.9 128 = 67 + 61
2 125.0 125 = 64 + 61
3 131.5 132 = 71 + 61
4 131.2 132 = 71 + 61
5 117.4 118 = 57 + 61
6 132.9 133 = 72 + 61
7 143.4 144 = 83 + 61
8 138.7 139 = 78 + 61
9 131.5 131 = 70 + 61
10 142.2 142 = 81 + 61
11 152.2 152 = 91 + 61
12 125.2 125 = 64 + 61
13 125.6 125 = 64 + 61
14 140.8 141 = 80 + 61
15 150.9 151 = 89 + 62
Tue Dec 3 22:43:31 PST 2013
Wed Dec 4 07:43:31 CET 2013
Now:
Tue Dec 3 23:01:20 PST 2013
Wed Dec 4 08:01:20 CET 2013
HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev
1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.3 7.0 1.1
2.|-- hos-tr1.juniper1.rz13.hetzner.de 900 900 0.0% 0.1 0.2 4.3 245.4 18.9
3.|-- core21.hetzner.de 900 900 0.0% 0.2 0.2 0.3 99.4 3.5
4.|-- core12.hetzner.de 900 900 0.0% 2.7 2.8 2.8 23.1 1.9
5.|-- juniper4.rz2.hetzner.de 900 899 0.1% 2.8 2.8 2.8 29.0 1.1
6.|-- te0-0-2-1.nr21.b040138-0.nue01.atlas.cogentco.com 900 900 0.0% 3.1 3.3 3.4 7.1 0.6
7.|-- te2-4.ccr01.nue01.atlas.cogentco.com 900 900 0.0% 3.1 6.6 25.9 244.4 51.0
| `|-- 154.25.0.13
8.|-- te0-2-0-3.ccr22.muc01.atlas.cogentco.com 900 900 0.0% 5.7 5.8 5.9 11.1 0.5
9.|-- be2229.ccr22.fra03.atlas.cogentco.com 900 900 0.0% 11.1 11.4 11.4 14.4 0.5
| `|-- 154.54.38.189
10.|-- te0-3-0-2.mpd22.ams03.atlas.cogentco.com 900 900 0.0% 17.7 17.9 17.9 49.1 1.1
| `|-- 154.54.39.193
11.|-- be2276.ccr22.lon13.atlas.cogentco.com 900 900 0.0% 25.3 27.4 27.4 33.2 1.3
| `|-- 154.54.37.106
| |-- 154.54.58.69
12.|-- te0-7-0-33.ccr22.bos01.atlas.cogentco.com 900 900 0.0% 93.1 93.9 93.9 98.8 1.2
| `|-- 154.54.44.189
| |-- 130.117.0.185
13.|-- te7-8.ccr02.alb02.atlas.cogentco.com 900 900 0.0% 96.8 119.9 128.8 364.2 56.9
| `|-- 154.54.27.153
14.|-- te7-8.ccr01.buf02.atlas.cogentco.com 900 900 0.0% 103.3 126.8 135.8 444.9 59.0
15.|-- te0-5-0-2.ccr21.cle04.atlas.cogentco.com 900 900 0.0% 108.6 109.5 109.5 113.5 1.3
16.|-- te3-2.ccr01.cmh02.atlas.cogentco.com 900 900 0.0% 111.1 134.8 143.5 467.0 59.4
17.|-- te4-7.ccr01.cvg02.atlas.cogentco.com 900 900 0.0% 113.8 122.5 126.2 467.9 39.6
18.|-- te3-3.ccr01.ind01.atlas.cogentco.com 900 899 0.1% 116.1 137.8 145.4 457.7 56.0
19.|-- 38.104.214.6 900 900 0.0% 115.8 118.3 118.5 206.0 8.1
20.|-- xe-0-1-0.11.rtr.ictc.indiana.gigapop.net 900 900 0.0% 115.9 116.7 116.7 128.4 1.5
21.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 900 900 0.0% 115.9 117.0 117.1 168.6 3.6
22.|-- ae-0.0.br2.bldc.net.uits.iu.edu 900 900 0.0% 116.8 117.7 117.7 130.8 1.8
23.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 900 899 0.1% 116.8 117.8 117.8 135.1 1.9
24.|-- m???c????????.indiana.edu 900 899 0.1% 117.1 117.9 117.9 122.0 1.2
Tue Dec 3 23:19:56 PST 2013
Wed Dec 4 08:19:56 CET 2013
0 120.1 120 = 57 + 63
1 117.3 117 = 54 + 63
2 120.3 120 = 57 + 63
3 117.3 117 = 54 + 63
4 117.4 118 = 54 + 64
5 117.3 118 = 54 + 64
6 120.0 120 = 57 + 63
7 117.3 118 = 54 + 64
8 117.3 118 = 54 + 64
9 117.6 118 = 54 + 64
10 117.3 118 = 54 + 64
11 117.4 118 = 54 + 64
12 117.4 117 = 53 + 64
13 118.6 118 = 55 + 63
14 117.5 117 = 53 + 64
15 117.3 117 = 53 + 64
Tue Dec 3 23:20:11 PST 2013
Wed Dec 4 08:20:11 CET 2013
Yet:
Cns# sh -c 'while (true); do date; env TZ=Europe/Berlin date; mtr --report{,-wide,-cycles=900} --interval 1 --order "SRL BGAWV" -4 ntp1.yycix.ca; date; env TZ=Europe/Berlin date; unbuffer hping --icmp-ts --count 16 ntp1.yycix.ca | perl -ne "if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {(\$s,\$p)=(\$1,\$2);} if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {(\$o,\$r,\$t)=(\$1,\$2,\$3);} if (/tsrtt=(\d+)/) {print \$s,qq/\t/,\$p,qq/\t/,\$1,qq/ = /,\$r-\$o,qq/ + /,\$o+\$1-\$t,qq/\n/;}"; done'
Tue Dec 3 23:32:37 PST 2013
Wed Dec 4 08:32:37 CET 2013
HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev
1.|-- static.??.???.4.46.clients.your-server.de 900 900 0.0% 0.5 0.9 1.2 9.7 1.2
2.|-- hos-tr4.juniper2.rz13.hetzner.de 900 900 0.0% 0.2 0.2 2.0 138.3 8.9
3.|-- core22.hetzner.de 900 900 0.0% 0.2 0.2 0.2 8.5 0.4
4.|-- core1.hetzner.de 900 900 0.0% 4.8 4.8 4.8 16.0 0.7
5.|-- juniper1.ffm.hetzner.de 900 900 0.0% 4.8 4.8 4.8 13.8 0.7
6.|-- 30gigabitethernet1-3.core1.ams1.he.net 900 900 0.0% 11.2 14.2 14.9 106.2 6.2
7.|-- 10ge1-4.core1.lon1.he.net 900 900 0.0% 18.0 20.5 20.8 55.6 4.1
8.|-- 10ge10-4.core1.nyc4.he.net 900 895 0.6% 86.4 119.3 120.0 158.6 13.1
9.|-- 10ge1-2.core1.tor1.he.net 900 896 0.4% 96.4 128.4 129.1 159.6 12.9
10.|-- 10ge3-1.core1.ywg1.he.net 900 895 0.6% 121.8 152.2 152.7 181.5 12.6
11.|-- 10ge1-1.core1.yyc1.he.net 900 888 1.3% 135.7 168.0 168.5 197.4 13.0
12.|-- sebo-systems-inc.gigabitethernet2-23.core1.yyc1.he.net 900 890 1.1% 138.4 168.2 168.6 190.9 12.5
13.|-- ??? 900 0 100.0 0.0 0.0 0.0 0.0 0.0
14.|-- ntp1.yycix.ca 900 893 0.8% 138.7 168.8 169.2 191.9 12.2
Tue Dec 3 23:49:07 PST 2013
Wed Dec 4 08:49:07 CET 2013
0 180.0 180 = 109 + 71
1 155.9 156 = 84 + 72
2 143.6 144 = 72 + 72
3 179.8 180 = 108 + 72
4 161.0 161 = 89 + 72
5 155.9 156 = 84 + 72
6 168.0 168 = 97 + 71
7 164.5 165 = 93 + 72
8 177.9 178 = 107 + 71
9 159.6 160 = 88 + 72
10 179.1 180 = 108 + 72
11 178.6 178 = 106 + 72
12 149.8 149 = 78 + 71
13 166.2 166 = 94 + 72
14 146.8 147 = 75 + 72
15 153.7 154 = 82 + 72
Tue Dec 3 23:49:22 PST 2013
Wed Dec 4 08:49:22 CET 2013
Cheers,
Constantine.
On 2013-W48-7 01:11 -0800, Constantine A. Murenin wrote:
> Anyhow, is this better?
>
> I now saw a 2% traffic loss this night at a random test time, and
> the 151ms avg rtt on this 114ms rtt route.
>
>
> Cns# date ; mtr --report{,-wide,-cycles=600} --interval 0.5 --order "SRL BGAWV" -4 ????c????????.indiana.edu ; date
> Sat Nov 30 23:17:13 PST 2013
> HOST: Cns??????? Snt Rcv Loss% Best Gmean Avg Wrst StDev
> 1.|-- static.??.???.4.46.clients.your-server.de 600 600 0.0% 0.5 1.0 1.3 4.6 1.1
> 2.|-- hos-tr1.juniper1.rz13.hetzner.de 600 600 0.0% 0.1 0.2 2.0 58.5 7.9
> 3.|-- core21.hetzner.de 600 600 0.0% 0.2 0.2 0.2 10.2 0.7
> 4.|-- core22.hetzner.de 600 600 0.0% 0.2 0.2 0.2 11.2 0.8
> 5.|-- core1.hetzner.de 600 600 0.0% 4.8 4.8 4.8 25.1 1.3
> 6.|-- juniper1.ffm.hetzner.de 600 600 0.0% 4.8 4.8 4.8 13.9 0.6
> 7.|-- 30gigabitethernet1-3.core1.ams1.he.net 600 595 0.8% 11.2 14.3 15.2 121.4 7.4
> 8.|-- 10gigabitethernet1-4.core1.lon1.he.net 600 600 0.0% 18.2 21.0 21.3 51.2 4.0
> 9.|-- 10gigabitethernet10-4.core1.nyc4.he.net 600 592 1.3% 86.9 125.9 126.4 160.7 10.6
> 10.|-- 100gigabitethernet7-2.core1.chi1.he.net 600 591 1.5% 106.6 145.1 145.4 190.9 10.5
> 11.|-- ??? 600 0 100.0 0.0 0.0 0.0 0.0 0.0
> 12.|-- et-11-0-0.945.rtr.ictc.indiana.gigapop.net 600 589 1.8% 114.3 148.9 149.2 167.9 9.1
> 13.|-- xe-0-3-0.11.br2.ictc.net.uits.iu.edu 600 589 1.8% 113.4 149.2 149.5 173.4 9.3
> 14.|-- ae-0.0.br2.bldc.net.uits.iu.edu 600 590 1.7% 114.5 150.2 150.5 175.6 9.3
> 15.|-- ae-10.0.cr3.bldc.net.uits.iu.edu 600 589 1.8% 114.3 150.5 150.8 181.0 9.1
> 16.|-- ????c????????.indiana.edu 600 589 1.8% 114.8 150.7 151.0 170.7 9.0
> Sat Nov 30 23:24:06 PST 2013
>
>
>
> The ICMP timestamp request/reply test still indicates that only
> one path is affected: the one from Europe to US over he.net.
>
>
> Cns# date ; unbuffer hping --icmp-ts --count 30 ????c????????.indiana.edu | \
> perl -ne 'if (/icmp_seq=(\d+) rtt=(\d+\.\d)/) {($s, $p) = ($1, $2);} \
> if (/ate=(\d+) Receive=(\d+) Transmit=(\d+)/) {($o, $r, $t) = ($1, $2, $3);} \
> if (/tsrtt=(\d+)/) { \
> print $s, "\t", $p, "\t", $1, " = ", $r - $o, " + ", $o + $1 - $t, "\n"; }'
> Sun Dec 1 00:55:46 PST 2013
> 0 151.3 151 = 91 + 60
> 1 154.2 154 = 93 + 61
> 2 127.8 127 = 67 + 60
> 3 123.6 123 = 63 + 60
> 4 136.9 137 = 76 + 61
> 5 149.6 149 = 89 + 60
> 6 147.4 147 = 87 + 60
> 7 133.5 133 = 73 + 60
> 8 152.2 152 = 92 + 60
> 9 137.3 137 = 77 + 60
> 10 143.7 144 = 84 + 60
> 11 124.5 124 = 64 + 60
> 12 141.4 141 = 81 + 60
> 13 118.0 118 = 58 + 60
> 14 153.6 154 = 94 + 60
> 15 137.7 138 = 78 + 60
> 16 119.9 120 = 60 + 60
> 17 130.6 131 = 71 + 60
> 18 144.6 145 = 85 + 60
> 19 138.8 139 = 79 + 60
> 20 155.7 156 = 96 + 60
> 21 128.8 129 = 69 + 60
> 22 153.0 153 = 93 + 60
> 23 146.5 147 = 87 + 60
> 24 137.2 138 = 77 + 61
> 25 153.3 154 = 94 + 60
> 26 146.3 147 = 87 + 60
> 27 150.1 151 = 91 + 60
> 28 150.5 150 = 90 + 60
> 29 143.5 143 = 83 + 60
>
>
>
> Cheers,
> Constantine.
More information about the NANOG
mailing list