Facility wide DR/Continuity

Wed Jun 3 14:37:40 UTC 2009

On Wed, Jun 3, 2009 at 8:09 AM, Drew Weaver<drew.weaver at thenap.com> wrote:
> I'm attempting to devise a method which will provide continuous
>operation of certain resources in the event of a disaster at a single facility.

Drew,

If you can afford it, stretch the LAN across the facilities via fiber
and rebuild the critical services as a load balanced active-active
cluster. Then a facility failure and a routine server failure are
identical and are handled by the load balancer. F5's if you like
commercial solutions, Linux LVS if you're partial to open source as I
am. Then make sure you have a Internet entry into each location with
BGP.

BTW, this tends to make maintenance easier too. Just remove servers
from the cluster when you need to work on them and add them back in
when you're done. Really reduces the off-hours maintenance windows.

This is how I did it when I worked at the DNC and it worked flawlessly.

If you can't afford the fiber or need to put the DR site too far away
for fiber to be practical, you can still build a network which
virtualizes your LAN. However, you then have to worry about issues
with the broadcast domain and traffic demand between the clustered
servers over the slower WAN.

It's doable. I've done it with VPNs over Internet T1's. But you better
have your developers on board early and and provide them with a
simulated environment so that they can get used to the idea of having
little bandwidth between the clustered servers.

On Wed, Jun 3, 2009 at 9:25 AM, Ricky Duman<rduman at internap.com> wrote:
> - Failover to backup servers using DNS (but may not be instant)

If your budget is more than a shoestring, save yourself some grief and
don't go down this road. Even with the TTLs set to 5 minutes, it takes
hours to get to two-nines recovery from a DNS change and months to get
to five-nines. The DNS protocol is designed to be able to recover
quickly but the applications which use it aren't. Like web browsers.
Google "DNS Pinning."

Regards,
Bill Herrin

-- 
William D. Herrin ................ herrin at dirtside.com  bill at herrin.us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004