30% packet loss between cox.net and hetzner.de, possibly at tinet.net
Constantine A. Murenin
mureninc at gmail.com
Sat Apr 6 23:20:42 UTC 2013
On 2013-W14-5 21:27 -0700, Constantine A. Murenin wrote:
> On 2013-W14-6 05:04 +0300, Denys Fedoryshchenko wrote:
> > On 2013-04-06 04:32, Constantine A. Murenin wrote:
> > >Hello,
> > >
> > >There has been at least a 25% packet loss between hetzner.de and
> > >cox.net
> > >in the last couple of hours.
> > >
> > >Tried contacting hetzner.de, but they said it's not on their network.
> > >This has already happened a couple of days ago, too (strangely, on
> > >April 1),
> > >but then was good for the rest of the week -- no problems whatsoever.
> > >
> > >I wouldn't really care about this, if not for ssh:
> > >it just doesn't work on such huge loss.
> > >
> > >No other routes or networks seem affected.
> > >
> > >Any advice?
> > >
> > Doesnt looks like tinet for me.
>
> Might have been eurorings.net, as your Amazon EC2 to Hetzner
> traceroute seemed to suggest?
>
>
> This loss was apparent even with their own main websites:
This is now happening again today, in a similar timeframe:
% mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" hetzner.de ; date
HOST: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev
4.|-- COX-68-12-10-121-static.coxinet.net 60 60 0.0% 8.8 12.3 12.9 43.4 5.4
5.|-- COX-68-12-8-132-static.coxinet.net 60 60 0.0% 11.4 14.5 14.6 27.0 2.5
6.|-- 68.1.5.161 60 60 0.0% 56.1 60.4 60.5 74.0 4.0
7.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 55.9 60.5 60.7 93.3 5.4
8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 150.1 154.0 154.1 167.0 2.9
9.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 147.8 152.3 152.5 201.1 7.1
10.|-- ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 143.9 148.0 148.1 174.0 4.6
11.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 147.4 160.6 163.4 336.0 36.2
12.|-- kpn-gw.hetzner.de 60 51 15.0% 167.3 172.5 172.5 184.3 3.5
13.|-- hos-bb2.juniper3.rz2.hetzner.de 60 60 0.0% 144.4 148.2 148.2 159.0 2.5
14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de 60 59 1.7% 145.5 150.1 150.1 157.8 3.0
15.|-- www.hetzner.de 60 47 21.7% 167.0 173.2 173.3 203.2 7.1
Sat 6 Apr 2013 15:14:00 PDT
% traceroute -w2 -I hetzner.de; date
traceroute to hetzner.de (213.133.107.227), 64 hops max, 60 byte packets
4 COX-68-12-10-121-static.coxinet.net (68.12.10.121) 22.012 ms 14.112 ms 11.230 ms
5 COX-68-12-8-132-static.coxinet.net (68.12.8.132) 23.021 ms 13.239 ms 12.400 ms
6 68.1.5.161 (68.1.5.161) 60.956 ms 59.738 ms 57.743 ms
7 nyk-s2-rou-1001.US.eurorings.net (134.222.248.13) 58.906 ms 60.571 ms 70.399 ms
8 nntr-s1-rou-1101.FR.eurorings.net (134.222.226.162) 152.597 ms 153.552 ms 150.250 ms
9 kehl-s2-rou-1103.DE.eurorings.net (134.222.227.121) 152.905 ms 149.469 ms 150.648 ms
10 ffm-s1-rou-1102.DE.eurorings.net (134.222.227.177) 148.114 ms 144.579 ms 147.114 ms
11 nbg-s1-rou-1001.DE.eurorings.net (134.222.227.118) 148.728 ms 150.950 ms 148.117 ms
12 kpn-gw.hetzner.de (134.222.107.21) 170.385 ms * 167.488 ms
13 hos-bb2.juniper3.rz2.hetzner.de (213.239.240.129) 147.476 ms 149.074 ms 144.824 ms
14 hos-tr4.ms-ex3k2.rz1.hetzner.de (213.239.193.243) 161.621 ms 148.504 ms 147.931 ms
15 * www.hetzner.de (213.133.107.227) 169.137 ms 176.884 ms
Sat 6 Apr 2013 15:28:32 PDT
Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" cox.net ; date
HOST: XXXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev
1.|-- static.33.203.4.46.clients.your-server.de 60 60 0.0% 0.6 1.2 1.5 3.8 1.1
2.|-- hos-tr1.juniper1.rz13.hetzner.de 60 60 0.0% 0.2 0.3 1.8 34.6 5.8
3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.2 5.2 77.2 10.9
4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.6 5.7 13.3 3.9
5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.1 4.7 13.9 2.9
6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 7.9 8.6 17.8 3.8
7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.9 9.8 17.1 4.3
8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.2 6.9 59.5 7.0
9.|-- xe-10-2-2.was14.ip4.tinet.net 60 52 13.3% 115.3 119.8 120.1 163.4 9.1
10.|-- cox-communications-gw.ip4.tinet.net 60 51 15.0% 119.5 123.7 124.0 175.8 10.3
11.|-- dukedsrj02-ge210.0.rd.at.cox.net 60 48 20.0% 132.4 135.6 135.7 165.3 5.8
12.|-- 68.1.15.238 60 47 21.7% 132.9 135.0 135.1 146.5 2.0
13.|-- 68.99.123.4 60 33 45.0% 135.1 136.7 136.7 139.1 1.1
14.|-- ww2.cox.com 60 50 16.7% 135.1 136.8 136.8 139.2 1.1
Sat Apr 6 15:13:49 PDT 2013
Cns# traceroute -w2 -I cox.net; date
traceroute to cox.net (68.99.123.161), 64 hops max, 60 byte packets
1 static.33.203.4.46.clients.your-server.de (46.4.203.33) 0.790 ms 3.658 ms 0.486 ms
2 hos-tr1.juniper1.rz13.hetzner.de (213.239.224.1) 0.244 ms 0.241 ms 0.251 ms
3 hos-bb2.juniper4.rz2.hetzner.de (213.239.240.138) 2.735 ms 2.772 ms 2.752 ms
4 r1nue1.core.init7.net (77.109.135.101) 2.811 ms 2.833 ms 2.769 ms
5 r1nue2.core.init7.net (77.109.140.154) 2.955 ms 2.928 ms 2.896 ms
6 r1fra2.core.init7.net (77.109.140.49) 14.891 ms 5.720 ms 13.291 ms
7 r1fra1.core.init7.net (77.109.128.137) 5.935 ms 7.804 ms 5.905 ms
8 xe-4-2-2.fra23.ip4.tinet.net (77.67.76.237) 5.885 ms 5.920 ms 5.912 ms
9 xe-10-2-2.was14.ip4.tinet.net (141.136.110.18) 115.127 ms 121.543 ms *
10 cox-communications-gw.ip4.tinet.net (77.67.79.234) 119.574 ms 119.245 ms 119.170 ms
11 dukedsrj02-ge210.0.rd.at.cox.net (68.1.1.123) 132.619 ms 132.431 ms 132.363 ms
12 68.1.15.238 (68.1.15.238) 132.718 ms 132.689 ms 132.599 ms
13 68.99.123.4 (68.99.123.4) 134.947 ms * 134.671 ms
14 * ww2.cox.com (68.99.123.161) 135.139 ms 135 ms
Sat Apr 6 15:44:12 PDT 2013
It seems like hetzner.de <-> Amazon EC2 connectivity is indeed
also affected, in addition to hetzner.de <-> cox.net:
Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" www.tarsnap.com ; date
2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 2.8 50.1 9.6
3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.5 5.9 51.2 10.4
4.|-- r1nue1.core.init7.net 60 60 0.0% 2.8 4.7 5.6 14.2 3.5
5.|-- r1nue2.core.init7.net 60 60 0.0% 2.9 4.3 5.2 13.2 3.8
6.|-- r1fra2.core.init7.net 60 60 0.0% 5.7 8.8 9.7 18.4 4.6
7.|-- r1fra1.core.init7.net 60 60 0.0% 5.9 8.2 9.0 24.7 4.3
8.|-- xe-4-2-2.fra23.ip4.tinet.net 60 60 0.0% 5.9 6.4 7.0 43.7 5.4
9.|-- xe-4-1-1.was14.ip4.tinet.net 60 45 25.0% 115.1 117.3 117.6 173.2 9.6
10.|-- vadata-gw.ip4.tinet.net 60 50 16.7% 117.0 117.6 117.7 130.0 2.1
11.|-- 72.21.220.31 60 50 16.7% 118.1 120.2 120.2 140.4 4.6
12.|-- 205.251.245.55 60 49 18.3% 120.2 121.1 121.1 132.0 2.1
13.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
14.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
15.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
16.|-- 216.182.224.129 60 54 10.0% 118.2 118.9 118.9 130.3 1.7
17.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
18.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
19.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
20.|-- ec2-23-21-149-109.compute-1.amazonaws.com 59 53 10.2% 118.6 119.4 119.4 139.8 2.9
Sat Apr 6 15:21:06 PDT 2013
On the other hand, comcast.net is not affected, att.net not affected,
he.net not affected, lots of other routes not affected.
Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" comcast.net ; date
2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 3.9 56.6 11.6
3.|-- hos-bb2.juniper8.rz1.hetzner.de 60 60 0.0% 2.8 2.9 3.1 24.9 2.9
4.|-- nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 3.0 3.6 7.3 177.0 24.2
5.|-- mchn-s1-rou-1021.DE.eurorings.net 60 60 0.0% 6.9 7.6 8.0 24.1 3.8
6.|-- mchn-s1-rou-1103.DE.eurorings.net 60 60 0.0% 91.8 93.7 93.9 133.0 6.7
7.|-- kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 93.9 95.8 96.1 134.2 7.3
8.|-- nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 91.8 94.2 94.3 127.6 5.3
9.|-- nyk-s2-rou-1021.US.eurorings.net 60 60 0.0% 92.1 95.7 96.0 128.3 8.2
10.|-- nyk-s2-rou-1001.US.eurorings.net 60 60 0.0% 91.7 93.2 93.7 188.9 12.6
11.|-- te-1-10-0-4-pe01.111eighthave.ny.ibone.comcast.net 60 60 0.0% 91.7 92.3 92.3 93.7 0.3
12.|-- pos-1-6-0-0-cr01.newyork.ny.ibone.comcast.net 60 60 0.0% 92.2 94.0 94.0 95.8 1.1
13.|-- he-0-0-0-0-cr01.350ecermak.il.ibone.comcast.net 60 60 0.0% 111.5 117.1 117.2 123.3 3.6
14.|-- pos-0-3-0-0-ar02.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.8 113.0 113.0 113.5 0.1
15.|-- te-0-3-0-0-ur04.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.8 113.0 113.0 113.3 0.1
16.|-- ge-0-1-0-0-ur03.northlake.il.ndcchgo.comcast.net 60 60 0.0% 112.9 113.0 113.0 113.3 0.1
17.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
Sat Apr 6 15:47:41 PDT 2013
Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" att.net ; date
2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.2 1.0 45.2 5.8
3.|-- hos-bb2.juniper4.rz2.hetzner.de 60 60 0.0% 2.7 3.7 8.0 86.2 17.5
4.|-- ae55.edge7.Frankfurt1.Level3.net 60 60 0.0% 7.2 7.9 8.9 57.8 7.9
5.|-- ??? 60 0 100.0 0.0 0.0 0.0 0.0 0.0
6.|-- ae-62-62.ebr2.Frankfurt1.Level3.net 60 60 0.0% 85.3 85.3 85.3 87.2 0.3
7.|-- ae-23-23.ebr2.London1.Level3.net 60 60 0.0% 85.3 85.3 85.3 85.4 0.0
8.|-- ae-43-43.ebr1.NewYork1.Level3.net 60 60 0.0% 85.4 85.5 85.5 88.6 0.4
9.|-- ae-71-71.csw2.NewYork1.Level3.net 60 60 0.0% 85.4 87.6 87.7 97.7 3.9
10.|-- ae-2-70.edge3.NewYork1.Level3.net 60 60 0.0% 85.5 87.5 88.0 154.7 10.9
11.|-- att-level3.newyork1.level3.net 60 60 0.0% 86.5 88.3 88.3 91.8 1.2
12.|-- cr2.n54ny.ip.att.net 60 60 0.0% 111.5 116.7 116.8 124.9 4.9
13.|-- cr2.wswdc.ip.att.net 60 60 0.0% 110.1 115.4 115.5 123.9 5.1
14.|-- cr1.attga.ip.att.net 60 60 0.0% 111.3 116.7 116.8 124.9 4.8
...
Sat Apr 6 15:36:40 PDT 2013
Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" he.net ; date
2.|-- hos-tr3.juniper2.rz13.hetzner.de 60 60 0.0% 0.2 0.3 4.2 71.3 13.9
3.|-- hos-bb2.juniper4.ffm.hetzner.de 60 60 0.0% 5.8 5.9 6.0 9.9 0.7
4.|-- 30gigabitethernet4-3.core1.fra1.he.net 60 60 0.0% 5.7 8.3 8.8 17.6 3.3
5.|-- 10gigabitethernet1-4.core1.par2.he.net 60 60 0.0% 15.1 18.1 18.5 26.0 3.7
6.|-- 10gigabitethernet7-1.core1.ash1.he.net 60 60 0.0% 93.1 95.9 95.9 105.7 3.5
7.|-- 10gigabitethernet11-1.core1.pao1.he.net 60 60 0.0% 163.4 167.5 167.6 177.0 4.1
8.|-- 10gigabitethernet1-2.core1.fmt1.he.net 60 60 0.0% 164.5 170.2 170.5 248.5 12.6
9.|-- he.net 60 59 1.7% 164.0 164.6 164.6 165.2 0.2
Sat Apr 6 15:52:46 PDT 2013
Although hetzner.de claims that this whole loss is outside of their own
network, I'm inclined to deduce that the loss might actually be
concentrated on their own KPN / eurorings.net router --
kpn-gw.hetzner.de (134.222.107.21), and perhaps occurs only
in one direction.
Although there is no traffic loss from he.net if you try to traceroute
the router itself (I'm not sure what that means, though, other than a
potential attack vector from exposing a router globally like that):
# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 134.222.107.21 ; date
HOST: XXXXXXXXX Snt Rcv Loss% Best Gmean Avg Wrst StDev
1. router1-fmt.linode.com 60 60 0.0% 0.5 1.5 4.8 70.2 10.6
2. 10gigabitethernet2-3.core1.fmt1.he.net 60 60 0.0% 0.6 1.0 3.5 72.1 11.9
3. 10gigabitethernet1-1.core1.pao1.he.net 60 60 0.0% 1.0 1.2 1.4 12.4 1.5
4. eqix-sv8.kpn.com 60 60 0.0% 1.8 2.5 5.9 77.9 15.2
5. chg-s1-rou-1041.US.eurorings.net 60 60 0.0% 161.5 167.1 167.3 197.9 7.5
6. ahbn-s1-rou-1041.US.eurorings.net 60 60 0.0% 71.5 74.0 74.3 108.9 7.5
7. nyk-s2-rou-1021.US.eurorings.net 60 60 0.0% 71.6 75.7 76.2 131.7 10.3
8. nntr-s1-rou-1101.FR.eurorings.net 60 60 0.0% 161.6 164.5 164.6 188.8 6.8
9. kehl-s2-rou-1103.DE.eurorings.net 60 60 0.0% 161.5 162.9 163.0 206.3 6.0
10. ffm-s1-rou-1102.DE.eurorings.net 60 60 0.0% 157.9 166.9 167.1 193.8 9.4
11. nbg-s1-rou-1001.DE.eurorings.net 60 60 0.0% 161.6 163.6 163.9 235.2 10.3
12. kpn-gw.hetzner.de 60 59 1.7% 161.3 161.9 161.9 167.5 1.0
Sat Apr 6 15:58:04 PDT 2013
li163-159:~# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 hetzner.de ; date
HOST: xxxxxxxxx Snt Rcv Loss% Best Gmean Avg Wrst StDev
1. router1-fmt.linode.com 60 60 0.0% 0.5 8.6 34.8 396.6 66.0
2. 10gigabitethernet2-3.core1.fmt1.he.net 60 60 0.0% 0.6 2.7 6.7 75.1 12.2
3. 10gigabitethernet1-1.core1.pao1.he.net 60 60 0.0% 0.9 2.2 3.8 22.1 4.3
4. 10gigabitethernet9-3.core1.ash1.he.net 60 60 0.0% 71.2 74.9 75.1 88.7 4.9
5. 10gigabitethernet1-2.core1.par2.he.net 60 60 0.0% 149.0 151.9 152.0 159.8 3.5
6. 10gigabitethernet4-4.core1.fra1.he.net 60 60 0.0% 158.2 161.3 161.4 169.1 3.8
7. decix-gw.hetzner.de 60 60 0.0% 158.6 161.2 161.4 209.3 8.9
8. hos-bb1.juniper3.rz2.hetzner.de 60 60 0.0% 161.8 173.2 174.0 220.2 18.4
9. hos-tr3.ms-ex3k2.rz1.hetzner.de 60 60 0.0% 162.6 163.7 163.7 166.0 1.1
10. www.hetzner.de 60 59 1.7% 161.8 162.4 162.4 163.1 0.3
Sat Apr 6 16:14:48 PDT 2013
I've been a fan of hetzner.de, but I think it's staggering that
they won't do anything about this huge and persistent packet loss.
Best regards,
Constantine.
More information about the NANOG
mailing list