Anyone know of a good InfiniBand vendor in the US?

Tom Ammon thomasammon at gmail.com
Wed Feb 20 22:25:18 UTC 2013


IPoIB looks more like an application than a network protocol to Infiniband.
The IB fabric doesn't have a concept of broadcast, so ARP works much
differently than it does in IPv4/ethernet world - basically an all-nodes
multicast group handles the distribution of ARP messages. That said, the ib
drivers that come with redhat/centos are pretty good, and you can always
download the official OFED drivers from the OFA at
https://www.openfabrics.org/linux-sources.html if the stuff in your linux
distribution is missing something.

I've set up IPoIB routers running 10G NICs on the ethernet side and QDR
HCAs on the IB side, using quagga to plug in to the rest of my OSPF
network, and it works fine. Basically you just need to set up quagga like
you would if you were going to turn a linux box into an ethernet router and
don't worry about the fact that it's actually IB on one side of the router
- your network statements, etc., in OSPF in quagga won't change at all.

You'll find that some things in IB have no equivalent to ethernet. For
example, if you want to have gateway redundancy for traffic exiting the IB
fabric, your first instinct will be to look for VRRP for IB, but you won't
find it, because of the ARP differences I talked about above. To get around
this you can set up linux-ha or some other type of heartbeat arrangement
and bring up a virtual IP on the active gateway, which can be shifted over
to the standby gateway when the ha scripts detect a problem. Some vendors
also have proprietary solutions to this problem but they tend to be
expensive.

So, I'd say, read up on quagga and give that a try, and I think you'll find
that as long as the IB drivers are up to snuff (the sminfo command returns
valid results, etc.) it'll pretty much just work for you. I'm also happy to
discuss more offline if you prefer.

Tom

Tom


On Tue, Feb 19, 2013 at 5:55 PM, Jon Lewis <jlewis at lewis.org> wrote:

> On Tue, 19 Feb 2013, Landon Stewart wrote:
>
>  Oh by vendor I mean VAR I guess.  Mostly I'm also wondering how an IB
>> network handles IPoIB and how one uses IB with a gateway to layer 3
>> Ethernet switches or edge routers.  If anyone has any resources that
>> provide details on how this works and how ethernet VLANs are handled I'd
>> appreciate it.
>>
>
> My limited IB experience has been that the IB switch acts much like a dumb
> ethernet switch, caring only about which IB hardware addresses are
> reachable via which port.  Routing between IPoIB and IP over ethernet can
> be done by any host with interfaces on both networks and IP forwarding
> enabled.  In our setups, we've used IPoIB, but with 1918 addresses and not
> routed beyond the IB network.
>
> ------------------------------**------------------------------**----------
>  Jon Lewis, MCP :)           |  I route
>  Senior Network Engineer     |  therefore you are
>  Atlantic Net                |
> _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
>
>


-- 
-----------------------------------------------------------------------------
Tom Ammon
Network Engineer
M: (801) 674-9273
tom at tomsbox.net
-----------------------------------------------------------------------------



More information about the NANOG mailing list