An Easy way to build a server cluster without top of rack switches (MEMO)

Mon Feb 16 03:23:41 UTC 2015

Hi Dan and ken.

I respect your great works.

Certainly, our scenario was network classics and it just does not "one size
fits all" network architecture.
Many people tried to built centralized and decentralized networks many
years ago, some guys output
implementation like this.

Interconnect of K computer (torus fusion)
https://www.fujitsu.com/global/Images/fujitsu-hpc-roadmap-beyond-petascale-computing.pdf

I agree with you point. Our approach have to do more simple way on physical
and logical
network engineering, and change the mindset, I think.
(e.g. network cabling procedure and troubleshooting and handling)

But, some guys need more cost effective server cluster environment and they
does't care
network latency like Low-End Web Hosting.

e.g. Intel Diversity of Server workloads http://bit.ly/1BgFH65 [JPG].

Now, Many people do not use Dijkstra and automaton theory on the server
side,
but it is great mechanism for network durability if they controlled.

The Ethernet NIC's bandwidth is increasing day by day, the cost is
 decreasing too.

I say again, our scenario is not "one size fits all" network architecture,
but we believe that something will happen for some guys works. ;-)

best regards,

--
Naoto MATSUMOTO

On Sat, Feb 14, 2015 at 7:08 AM, Dan Eckert <daniel.eckert at microsoft.com>
wrote:

> I'm having a hard time seeing how this reduces cable costs or increases
> network durability.  Each individual server is well connected to 3-4 other
> servers in the rack, but the rack still only has two uplinks.  For many
> servers in the rack you're adding 3-4 routing hops between an end node and
> the rack uplink.
>
> Additionally, with only 2 external links tied to 2 specific nodes, you
> introduce more risks.  If one of the uplink nodes fails, you've got to
> re-route all of the nodes that were using it as the shortest path to now
> exit through the other uplink node -- the worst case in the example then
> increases from the original 4-hops-to-exit to now 7-hops-to-exit.
>
> As far as cable costs go, you might have slightly shorter cables, but far
> more complex wiring pattern -- so in essence you're trading off a small
> amount of cable cost for a higher amount of installation and
> troubleshooting cost.
>
> Also, using this layout, you dramatically reduce the effective bandwidth
> available between devices, since per-device links now have to be used for
> backhaul/transport in addition to device-specific traffic.
>
> Finally, you have to manage per-server routing service configurations to
> make this work -- more points of failure and increased
> setup/troubleshooting cost.  In a ToR switch scenario, you do one config on
> one switch, plug in the cables, and you're done -- problems happen, you go
> to the one switch, not chasing a needle through a haystack of
> interconnected servers.
>
> If your RU count is worth more than the combination of increased
> installation, server configuration, troubleshooting, latency, and capacity
> costs, then this is a good solution.  Either way, it's a neat idea and a
> fun thought experiment to work through.
>
> Thanks!
> Dan
>
>
> -----Original Message-----
> From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of NAOTO MATSUMOTO
> Sent: Wednesday, February 11, 2015 11:32 PM
> To: nanog at nanog.org
> Subject: FYI: An Easy way to build a server cluster without top of rack
> switches (MEMO)
>
> Hi all!
>
> We wrote up TIPS memo "an easy way to build a server cluster without top
> of rack switches" concept.
>
> This model have a reduce switches and cables costs and high network
> durability by lightweight and simple configuration.
>
> if you interest in, please try to do yourself this concept  ;-)
>
>
> An Easy way to build a server cluster without top of rack switches (MEMO)
> http://slidesha.re/1EduYXM
>
>
> Best regards,
> --
> Naoto MATSUMOTO
>