The trek across Canada

Fri Nov 6 07:56:26 UTC 1998

> I'm looking for a cost-effective, highly reliable, high bandwidth, low
> latency solution for a link in Canada between Vancouver and Toronto. 

  Tall order.

> I have a couple of ideas:
> 
> 1)  Droping some type of high-speed interface into the frame-relay cloud
> of a third party network like AT&T, Sprint, MetroNet, or whoever provides
> this type of service. I think this would definetly be a scalable solution
> as it would allow us to put POPs anywhere we like in between and it would
> be as easy as running a circuit back into the frame cloud.  

  You also should consider the availability of access to include
  proximity to the frame switch, comprising an access point into
  the frame cloud.

  If the nearest Frame Access point is 100 km away, your cost
  of access is likely significantly higher than if you are
  close to the switch.

  Some NSPs provide subsidization of the loop back to the L2/L3 NE
  point, but most do not.

> The only
> issues with this is a) What are these networks like?  

  Generally they are comprised of a sparse mesh of FR/ATM switches,
  analagous to the backbones of most ISPs, w/ less NE redundancy,
  and more hops.

> Are they reliable? 

  Relative to what?  The uptime on most FR/ATM switches is probably
  much higher than most IP routers.

  The micro level reliability of the FR/ATM networks is typically
  1 - 2 notches above those of typical IP routers.  They are
  also typically less dynamic and more 'rigid'.

  The stdev on the quality tends to vary significantly, as BB
  issues on the L2 FR NSP cause PVCs to reroute, introducing
  step-wise latency/loss changes.

  From my experience, these seem to be much less pronounced on
  native IP networks, than on IP connectivity via the 3rd party
  L2 FR/ATM provider.

> Are they fast? 

  Most L2 FR networks are at DS3 speeds.  Most ATM neworks are
  hybrid DS3/OC-12.  To the best of my knowledge noone has
  yet deployed OC-48.

  Many/most L2 providers [ATM/FR] offer FR and ATM to customers, and
  use ATM as a backbone transport mechanism for FR.  This is done
  by 'emulating/transporting' FR PVCs over ATM PVCs.

> Is their a risk of them not being able to provide us with
> the bandwidth we paid for without performance loss due to lack of capacity
> (QoS)?

  Yes.

  It is common for the L2 FR/ATM NSP to oversubscribe their
  network, above and beyond that 'promised' w/ SCR/PCR/CIR
  contracts.

> 2)  Assuming we buy our own fibre:  

  It is unlikely you would purchase your own glass between
  cities, the cost would most likely be many millions of 
  dollars $US.

  This presents an interesting market system, where the supply is
  quite controlled by folks who have bought up Rights-Of-Way [ROW].
  The ROW tends to be tremendously expensive, due to the expanse of 
  distance, tremendous growth in DWDM technology==high value of
  ROW, and municipalitys being reticent to allow carriers to dig
  up streets.

  I'll assume for info below that you mean purchase DIM capacity
  or purchase SONET circuits from point A to Z.

> Do we run a direct link between
> Toronto and Vancouver directly or drop a POP in between; somewhere in
> Saskatchewan or Manitoba and running fibre out of there to each?  

  If you have business only in the cities at the end, [Toronto
  and Vancouver] then putting a POP in the midle is unlikely
  to help you, as it just adds a point of potential problems.

  If you have business in the middle cities, or semi-uniformly
  distributed across the country, building POPs across the
  country is a good thing.

  As you consider building a densely meshed network, you are
  likely to find that putting an L3 POP in the middle of the
  networks allows you to make a 'T' point or 'intersection' point
  where you can bring in additional bandwidth to the backbone
  fabric.  This will allow you to cut costs.  

> I may
> like the direct route from A to B but how pricy is that and how scalable
> is that?  

  Typical circuits are measured in '$/DS0s/mile'  A T3 is 28*24
  DS0s, typical pricing is in the ballpark of ~2 cents/ds0 mile, or
  the ballpark of $10/mile for a DS3, plus a 'termination' charge
  for each node intersecting the circuit.

  Typically adding nodes to the circuit does not significantly
  increase the cost; ie; adding manitoba or winchester does not
  change your per-mile costing, only add a relatively small fixed
  cost to the monthly circuit charge.

> If we plan do put POPs into some some cities along the way we'd
> have to run pipes from those POPs back into either Toronto of Vancouver. 

  Right.

> If we go the route of A to B to C that would probably be a bit smarter as
> the haul from POP to POP wouldn't be so large. 

  Right.

> Is it common practice for the National ISP to lay their own pipe or is it
> more common practice to obtain the link from a third party?  

  It is most common for them to be purchased and use their parent
  company's fiber :-)

  I would dichotomize NSPs into 2 categories - facilities based and
  non facilities based.

  Facilities based NSPs own their own fiber, and usually have
  75-100% of the backbone circuits on bandwidth owned by their
  company.

  Non Facilities based NSPs by definition obtain their bandwidth
  from a third-party.

  I would estimate that 95% of NSPs are non-facilities based.

  I would estimate that 70-90% of 'Internet Traffic' in the US
  travels over facilities based NSPs.

> How do the
> UUNets, and BBNPlanets do it?  

  You'll have to ask them.

  Most people would agree that the current economic models forbid an
  NSP from being successful on their own long term as a nationwide 
  NSP unless they are facilities based.

> They are not telcos so laying a pipe is not
> as easily justified.  

  Your two examples actually are facilities based, UUNET=WCOM;
  BBNPlanet=GTE.

> Do they buy fibre between A and B or do they even go
> through some sort of telco-like carrier?!

  Both.  And for additional confusion, most (facility based) carriers 
  enjoy incestuous fiber swaps and cross-leasing of capacity.

> If anyone has any ideas, opinions, experiences, or something to sell me 
> I'd love to hear from you!

  I've got lots of things to sell you, but this is not the forum :-)

  I hope this perspective helps you and perhaps others.

  -alan