vyatta for bgp

Mon Sep 26 20:22:25 UTC 2011

There are a lot of variables that would skew numbers in favor of using
FOSS on commodity hardware in our situation, that wouldn't necessarily
apply to others.  Primarily because these are used to provide services
that are in part funded through the federal E-rate program, and need
to comply with restrictions such as CIPA.

For example, we moved from centralized web filtering using WCCP and
racks of proxy servers, to pushing that service out to the edge.  That
move alone provided more savings than the hardware cost of the
project, so we actually made a net profit from the move in our
situation.

Not sure that would easily apply to anyone.

As for the OpEx and CapEx v. traditional players...

The units are engineered so that they run the entire OS on a RAM disk;
so configuration management is much like what you would find with a
traditional router (only saved configuration survives a reboot, etc --
think of it like a live distribution with controlled persistence).

A physical disk is used for logging, but does not take out the system
upon failure (we've had maybe 3 disk failures that turned out to be
thermal conditions of where equipment was installed -- boiler rooms --
and service was maintained until we had a technician out to swap the
unit).  So operationally, they've been pretty much equivalent of a
Cisco solution and we haven't seen much of an increase in activity
aside from supporting the extra services that weren't previously
available.

The skill set is a little different though.  Having a strong
understanding of the internals of a Linux system along side
traditional networking skills is a must if you go in this direction.

For us, the ability to have more tools to poke at the state of the
system and troubleshoot issues (such as performing packet captures
directly on the device) has been invaluable.  It has allowed us to
track down issues (such as TCP window scaling problems with unnamed
cloud services and their incorrectly configured load balancers)
remotely that would have required on-site capture in the past.

It's also provided us with the flexibility to quickly implement
operational changes as we see a need, such as implementing automatic
nightly backup of configurations to our central servers (using a
simple CRON job), or rolling out scripted changes.

Using an off-the-shelf distribution of Linux and a FOSS routing
package will probably not do the trick for you.  If you take the time
to build a custom distribution that only has what you need; makes use
of known stable package versions, and is engineered to function as a
widely-deployed unit (configuration management, logging, etc) that is
where the savings will come in, because you won't need to see the
significant increase in OpEx that opponents usually point to.  We were
debating on if we should do that in-house or not.  I think if you're
talking about 1000 units then in makes sense to try in-house, on a
smaller scale you really want to find a partner that can engineer the
system for you.

Vyatta looks like it's addressed a lot of the issues it needs to --
though I've never used it in production -- but I would still like to
see more from them in tuning the OS to function better as a router and
less like a server.  Last time I checked they didn't seem to touch
much except setting Linux to allow forwarding.  I'm optimistic though.

Now we just need Intel to step up with some ASICs and open source
drivers that could be plugged into Linux. (On a side note, we make use
of some SFP PCI-X cards for our direct optical connected sites to save
money there too; working well with up to ZX SFPs).

On Mon, Sep 26, 2011 at 3:20 PM, Scott Weeks <surfer at mauigateway.com> wrote:
> --- rps at maine.edu wrote:
> From: Ray Soucy <rps at maine.edu>
>
> We service most of the state's public schools and libraries (about
> 1000).  Historically the CPE of choice was a small Cisco ISR (1600,
> 1700, 1800, and 1900 most recently).  As bandwidth levels went up, and
> Ethernet-based transport services became available, we started looking
> and leveraging FOSS on commodity hardware to lower costs and move
> services to the edge.  Right now we have about 100 of the bigger
> school districts being services by a Linux-based appliance running
> XORP for its routing engine (we would have tried Quagga, but they
> don't support multicast routing yet, nor does Vyatta).
>
> It's been a learning experience.  Most of the problems we ran into
> have been resolved by tuning the kernel parameters to act more like a
> router than a desktop or server.  XORP itself has had a rocky ride
> since we started, so the stability of the project has also been a
> concern.  Thankfully it is seeing somewhat active development again.
> I will note that XORP is very touchy about how it's configured; if you
> have well tested configuration templates it's fine, but it's very easy
> to get it into a crashing state based on something as little the order
> of configuration directives.  For the most part once it's running it's
> stable.
> --------------------------------------------------------------------
>
>
>
> After roll-out and after a time in steady-state operation did you do an analysis of human and hardware/software costs (as well as service to end sites, such as outages that might not have happened with normal routers and LAN switches) to see if you actually saved money?
>
> scott
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

-- 
Ray Soucy

Epic Communications Specialist

Phone: +1 (207) 561-3526

Networkmaine, a Unit of the University of Maine System
http://www.networkmaine.net/