So I upgraded the homelab from Infinihost III gear to 40G QDR HBAs, Switch and cables.
- IBM Voltaire Infiniband 3036
- QLogic QLE7340 40GB HBAs
I'm using this fabric for a Gluster and oVirt, on Fedora 23, using the Fedora drivers (I wasn't able to compile OFED successfully).
First question, datagram or connected mode? I can't seem to modify the switch for a higher MTU than 2044 but certain gluster docs suggest connected mode, which brings the MTU to 65520.
Here is some ibstat/link information from a node, which all looks cool, but 4 of 5 links are at 20G, while one is at 40G? https://paste.fedoraproject.org/404408/
Now on the switch, which I'm using as the subnet manager, all the links fails. https://paste.fedoraproject.org/404410/
On spidey, here is some weirdness in dmesg, I'm quite sure what it means besides of the connected mode message. https://paste.fedoraproject.org/404411/
When using RDMA, it will connect with no issues but it seems like it runs out of memory or unable to create new connections and resets itself, browsing the tuning parameters from Mellanox, I haven't been able to stabalize the cards via RDMA, TCP is okay(ish). So things to check, things to look for, suggestions, thoughts, I'm at a lost, thanks.
[link] [comments]