Scaling from home broom closet to multisite "home" data center/WAN network on a budget

Tue Oct 14 22:17:20 UTC 2014

Hi everybody,

It's been a long time since I've kicked up a new thread here on ye ol 
nanog.

Recently I've been putting some serious thought into home "budget" data 
centers. What started out as a little router/switch/virt server lab by 
me/myself/I in 2008, has turned into a multisite (7 points of presence 
spread across two states), multi rack (223U, 10 racks), multi carrier 
(time warner, suddenlink, att dsl, att uverse, blended bandwidth) affair 
today.

I've realized that my friends and I may need to approach things a bit 
more..... professionally/seriously/circumspectly (is that a word?).

We run OSPF (and soon i/eBGP including out to amprnet) between all the 
sites, have a multi vendor (cisco/dell/pfsense/mikrotik/linux) l3/l2 
network. All the gear has been sourced from flea markets/ebay etc.

Anyone else on here have a similar situation? How do you deal with it? 
(More details/questions after the jump)

I mean my little lab is not quite as .... *ahem* involved as mr morris 
home data center
http://smorris.uber-geek.net/lab.htm

but it's also just a whee bit bigger then the typical /r/homelab posts.

https://commons.thefnf.org/index.php/FNF_Lab << needs a bit more work to 
reflect current reality, but you can get the general idea)

So.... how to home/small (multisite) business datacenter/network on 
budgetz? Not so much the software side of things (networking/operating 
system etc), but more the actual layer 1 stuff (and of course software 
to manage all that).

Things like:

Power

* Any good resources for wiring optimization/layout/dos/dont's? (like 
say how to hook up surge protector -> ups -> pdu -> gear ?)

* Network UPS tools is what we are considering "betting on" for managing 
our multi vendor (APC/Dell/Cyclades) PDU/UPS setup across all sites. 
(The joys of sourcing from flea markets). Does anything better exist? 
Any issues with running this across a WAN? Any tips for tying it into 
LDAP? (We have a large amount of lab gear that we wish to make available 
to the public for use by reservation, we don't want them being able to 
turn off our production gear) :)

Temperature / Environmental monitoring

* I've found the temper USB sensors to be perfect for per rack 
temperature statistics. I believe they will do humidity as well. Any 
other good environmental sensors for cheap? I know about PacketFlux 
which is cool for an all in one solution. (I'd love to use netbotz or 
something, but those are pricy).

Cooling

* BTU calculation resources?
* Cooling sizing?
* How hot can gear really run?
* Is per rack cooling/airflow via floor fans necessary?

For now we all just have room AC cooling the gear. So far all seems 
well, curious if folks have done stuff like pipe AC output directly into 
a cabinet etc?

Noise dampening

Any good room partitions or other solutions for noise cancelling? We've 
got combination of closed cabs and open racks, so a rack based solution 
won't quite cut it.

Also what about insurance?

Probably a thousand other questions/comments/ideas/suggestions, but I 
think this should kick off a great discussion!

It's funny how it kind of all just combines/grows and next thing you 
know, you've got yourself a whole little internet as it were. :)