An Easy way to build a server cluster without top of rack switches (MEMO)

Fri Feb 13 22:08:21 UTC 2015

I'm having a hard time seeing how this reduces cable costs or increases network durability.  Each individual server is well connected to 3-4 other servers in the rack, but the rack still only has two uplinks.  For many servers in the rack you're adding 3-4 routing hops between an end node and the rack uplink.

Additionally, with only 2 external links tied to 2 specific nodes, you introduce more risks.  If one of the uplink nodes fails, you've got to re-route all of the nodes that were using it as the shortest path to now exit through the other uplink node -- the worst case in the example then increases from the original 4-hops-to-exit to now 7-hops-to-exit.

As far as cable costs go, you might have slightly shorter cables, but far more complex wiring pattern -- so in essence you're trading off a small amount of cable cost for a higher amount of installation and troubleshooting cost.

Also, using this layout, you dramatically reduce the effective bandwidth available between devices, since per-device links now have to be used for backhaul/transport in addition to device-specific traffic.

Finally, you have to manage per-server routing service configurations to make this work -- more points of failure and increased setup/troubleshooting cost.  In a ToR switch scenario, you do one config on one switch, plug in the cables, and you're done -- problems happen, you go to the one switch, not chasing a needle through a haystack of interconnected servers.

If your RU count is worth more than the combination of increased installation, server configuration, troubleshooting, latency, and capacity costs, then this is a good solution.  Either way, it's a neat idea and a fun thought experiment to work through.

Thanks!
Dan

-----Original Message-----
From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of NAOTO MATSUMOTO
Sent: Wednesday, February 11, 2015 11:32 PM
To: nanog at nanog.org
Subject: FYI: An Easy way to build a server cluster without top of rack switches (MEMO)

Hi all!

We wrote up TIPS memo "an easy way to build a server cluster without top of rack switches" concept.

This model have a reduce switches and cables costs and high network durability by lightweight and simple configuration.

if you interest in, please try to do yourself this concept  ;-)

An Easy way to build a server cluster without top of rack switches (MEMO) http://slidesha.re/1EduYXM

Best regards,
--
Naoto MATSUMOTO