Anyone know of a good InfiniBand vendor in the US?

Peter Phaal peter.phaal at gmail.com
Thu Feb 21 21:04:11 UTC 2013


I wanted to bring attention to the following draft proposal from
Mellanox to export traffic information from InfiniBand switches:

http://sflow.org/draft_sflow_infiniband.txt

If you are an InfiniBand user, this is a great opportunity to think
about the types of metrics that you woud want from your switches in
order to better understand performance. The operational sensibility
that the NANOG audience brings is particularly valuable.

Comments on the proposal are welcome on the sFlow discussion group:

http://groups.google.com/group/sflow

On Wed, Feb 20, 2013 at 2:25 PM, Tom Ammon <thomasammon at gmail.com> wrote:
> IPoIB looks more like an application than a network protocol to Infiniband.
> The IB fabric doesn't have a concept of broadcast, so ARP works much
> differently than it does in IPv4/ethernet world - basically an all-nodes
> multicast group handles the distribution of ARP messages. That said, the ib
> drivers that come with redhat/centos are pretty good, and you can always
> download the official OFED drivers from the OFA at
> https://www.openfabrics.org/linux-sources.html if the stuff in your linux
> distribution is missing something.
>
> I've set up IPoIB routers running 10G NICs on the ethernet side and QDR
> HCAs on the IB side, using quagga to plug in to the rest of my OSPF
> network, and it works fine. Basically you just need to set up quagga like
> you would if you were going to turn a linux box into an ethernet router and
> don't worry about the fact that it's actually IB on one side of the router
> - your network statements, etc., in OSPF in quagga won't change at all.
>
> You'll find that some things in IB have no equivalent to ethernet. For
> example, if you want to have gateway redundancy for traffic exiting the IB
> fabric, your first instinct will be to look for VRRP for IB, but you won't
> find it, because of the ARP differences I talked about above. To get around
> this you can set up linux-ha or some other type of heartbeat arrangement
> and bring up a virtual IP on the active gateway, which can be shifted over
> to the standby gateway when the ha scripts detect a problem. Some vendors
> also have proprietary solutions to this problem but they tend to be
> expensive.
>
> So, I'd say, read up on quagga and give that a try, and I think you'll find
> that as long as the IB drivers are up to snuff (the sminfo command returns
> valid results, etc.) it'll pretty much just work for you. I'm also happy to
> discuss more offline if you prefer.
>
> Tom
>
> Tom
>
>
> On Tue, Feb 19, 2013 at 5:55 PM, Jon Lewis <jlewis at lewis.org> wrote:
>
>> On Tue, 19 Feb 2013, Landon Stewart wrote:
>>
>>  Oh by vendor I mean VAR I guess.  Mostly I'm also wondering how an IB
>>> network handles IPoIB and how one uses IB with a gateway to layer 3
>>> Ethernet switches or edge routers.  If anyone has any resources that
>>> provide details on how this works and how ethernet VLANs are handled I'd
>>> appreciate it.
>>>
>>
>> My limited IB experience has been that the IB switch acts much like a dumb
>> ethernet switch, caring only about which IB hardware addresses are
>> reachable via which port.  Routing between IPoIB and IP over ethernet can
>> be done by any host with interfaces on both networks and IP forwarding
>> enabled.  In our setups, we've used IPoIB, but with 1918 addresses and not
>> routed beyond the IB network.
>>
>> ------------------------------**------------------------------**----------
>>  Jon Lewis, MCP :)           |  I route
>>  Senior Network Engineer     |  therefore you are
>>  Atlantic Net                |
>> _________ http://www.lewis.org/~jlewis/**pgp<http://www.lewis.org/~jlewis/pgp>for PGP public key_________
>>
>>
>
>
> --
> -----------------------------------------------------------------------------
> Tom Ammon
> Network Engineer
> M: (801) 674-9273
> tom at tomsbox.net
> -----------------------------------------------------------------------------




More information about the NANOG mailing list