Cogent <-> Verizon peering congestion

Wed Feb 5 16:04:20 UTC 2014

We've been having trouble with congestion between Verizon and Cogent in Chicago since May 2013.  We had to move some traffic off of our Verizon connection to get around it.  Verizon has apparently had an internal ticket open for the problem since February 2013.  Their response in August 2013 was "Cogent is designing an upgrade to turn up the bandwidth".  Their response a few weeks ago was "Verizon is working with Cogent on increasing more capacity there is no deadline of completion, issue ongoing since 2013 possible completion in 2014".  In other words, they're blaming Cogent.

Ben Bridges
SpringNet

-----Original Message-----
From: Robert Glover [mailto:robertg at garlic.com] 
Sent: Tuesday, February 04, 2014 5:44 PM
To: nanog at nanog.org
Subject: Cogent <-> Verizon peering congestion

Hello,

For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon

  Host Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. router.garlic.com 0.0%    29    0.3   6.1   0.2 160.6  29.7
  2. vl203.mag03.sfo01.atlas.cogentco.com 0.0%    29    2.2   8.1   2.1 
161.1  29.5
  3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0%    29    2.9   2.7   
2.4   3.6   0.2
  4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0%    29    4.1   4.0   
3.7   4.8   0.2
  5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0%    29    4.5   4.7   
4.3   5.5   0.3
  6. verizon.sjc03.atlas.cogentco.com 22.2%    28  169.3 171.5 168.1 
193.5   6.9
  7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0%    28  205.8 180.6 
171.6 271.6  24.8
  8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3%    28  172.3 177.5 
171.7 250.8  18.3
  9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0%    28  197.9 
197.6 195.5 199.2   0.8

We have smokeping's from our side showing 30%+ packet loss from us
(AS4307) to Verizon.

All I have gotten from Cogent is a canned response:

---
The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon.  While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints.  Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements.  
Our peering engineers will continue to address this on a daily basis until resolved.
---

It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc.

Has anyone else been experiencing these issues?  Or does anyone have more information that what Cogent provided me in their canned statement?

-Bobby