Lossy cogent p2p experiences?

Thu Aug 31 20:51:11 UTC 2023

Cogent has asked many people NOT to purchase their ethernet private circuit
point to point service unless they can guarantee that you won't move any
single flow of greater than 2 Gbps. This works fine as long as the service
is used mostly for mixed IP traffic like a bunch of randomly mixed
customers together.

What you are trying to do is probably against the guidelines their
engineering group has given them for what they can sell now.

This is a known weird limitation with Cogent's private circuit service.

The best working theory that several people I know in the neteng community
have come up with is because Cogent does not want to adversely impact all
other customers on their router in some sites, where the site's upstreams
and links to neighboring POPs are implemented as something like 4 x 10
Gbps. In places where they have not upgraded that specific router to a full
100 Gbps upstream. Moving large flows >2Gbps could result in flat topping a
traffic chart on just 1 of those 10Gbps circuits.

On Thu, Aug 31, 2023 at 10:04 AM David Hubbard <
dhubbard at dino.hostasaurus.com> wrote:

> Hi all, curious if anyone who has used Cogent as a point to point provider
> has gone through packet loss issues with them and were able to successfully
> resolve?  I’ve got a non-rate-limited 10gig circuit between two geographic
> locations that have about 52ms of latency.  Mine is set up to support both
> jumbo frames and vlan tagging.  I do know Cogent packetizes these circuits,
> so they’re not like waves, and that the expected single session TCP
> performance may be limited to a few gbit/sec, but I should otherwise be
> able to fully utilize the circuit given enough flows.
>
>
>
> Circuit went live earlier this year, had zero issues with it.  Testing
> with common tools like iperf would allow several gbit/sec of TCP traffic
> using single flows, even without an optimized TCP stack.  Using parallel
> flows or UDP we could easily get close to wire speed.  Starting about ten
> weeks ago we had a significant slowdown, to even complete failure, of
> bursty data replication tasks between equipment that was using this
> circuit.  Rounds of testing demonstrate that new flows often experience
> significant initial packet loss of several thousand packets, and will then
> have ongoing lesser packet loss every five to ten seconds after that.
> There are times we can’t do better than 50 Mbit/sec, but it’s rare to
> achieve gigabit most of the time unless we do a bunch of streams with a lot
> of tuning.  UDP we also see the loss, but can still push many gigabits
> through with one sender, or wire speed with several nodes.
>
>
>
> For equipment which doesn’t use a tunable TCP stack, such as storage
> arrays or vmware, the retransmits completely ruin performance or may result
> in ongoing failure we can’t overcome.
>
>
>
> Cogent support has been about as bad as you can get.  Everything is great,
> clean your fiber, iperf isn’t a good test, install a physical loop oh wait
> we don’t want that so go pull it back off, new updates come at three to
> seven day intervals, etc.  If the performance had never been good to begin
> with I’d have just attributed this to their circuits, but since it worked
> until late June, I know something has changed.  I’m hoping someone else has
> run into this and maybe knows of some hints I could give them to
> investigate.  To me it sounds like there’s a rate limiter / policer defined
> somewhere in the circuit, or an overloaded interface/device we’re forced to
> traverse, but they assure me this is not the case and claim to have
> destroyed and rebuilt the logical circuit.
>
>
>
> Thanks!
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20230831/7b0b9341/attachment.html>