Providing service for geographically disparate fiber SANs ... Level3 ?

Mon Apr 26 11:35:22 UTC 2004

Joe Schmoe,

At 01:19 PM 26/04/2004, Joe Schmoe wrote:
[..]
>So the question is, if you need to build a fiber SAN
>across several datacenters, do you have the ability to
>have a real fiber connection between all of them, or
>is it a virtual connection, where your fiber goes to
>their routers, which goes to fiber between their
>routers, etc., making several hops between each node
>of your SAN ?

firstly, if you're talking "SAN" as in "Fibre Channel", note that it won't 
go via "routers"; you'll find that FC needs a native transport that isn't 
encapsulated into IP (i.e. WDM or SONET transport).
note that this may or may not be a good idea: Fibre Channel has lots of 
inherent limitations as distances increases.  at the most basic level, 
there is something called Buffer Credits, which is analogous to Admission 
Control.  once you've exhausted the BB_Credits, your maximum throughput 
will drop significantly.  rule-of-thumb based on 
average-frame-size-distribution on FC is ~1 BB_Credit per kilometre @ 2G FC 
and ~1 BB_Credit per 2km @ 1G FC.

if you DO wish to transport FC inside Fibre Channel, there are two paths to 
choose from: FCIP (Fibre Channel over IP) or iFCP.
i'd recommend the former, but note that i will be biased in saying that -- 
i 'do' storage at Cisco/Andiamo as my day job.

note that BB_Credits will likely not even end up being your bottleneck, but 
rather the fact that there are large chunks of both applications, 
filesystems and even SCSI that are simply synchronous in nature and slow 
down dramatically as latency increases.
there are some modern techniques that can be used to counteract those (e.g. 
spoofing the SCSI R2T, reducing a SCSI Write from 2 round-trips to one), 
and even spoofing an entire SCSI Write locally (effectively turning sync 
into async), and you may want to add both compression and/or encryption to 
your storage traffic also...

>This seems "bad" to me, because a router problem
>between the two nodes would essentially "break" the
>SAN - much like breaking a hardware RAID array by
>disconnecting a drive.

note that with FCIP, it is possible to engineer traffic in a way that 
failure of router/nodes won't cause a fabric disruption.
FCIP is FC encapsulated over TCP, so one can engineer parallel FCIP links 
(in a port-channel for load-balancing if you wish) to go via separate 
physical paths & across diverse carriers if you wish.

note that it is important to divorce the concept of what is something like 
'mirroring' from 'replication'.
it would be unlikely that you'd ever want to do 'RAID' across a 
geographically-dispersed SAN but its likely that you'd want to do 
'replication'.

>Comments ?  How are people, from the point of view of
>the network, building fiber SANs, and what providers
>(L3 ?) are they using ?

Sprint did some testing.  some details of that are at 
<http://www.cisco.com/warp/public/cc/pd/ps4159/ps4358/prodlit/sstcs_wp.pdf>

i'm happy to talk more off-list, lest this violate nanog's terms of conditions.

cheers,

lincoln.