I have three hosts connected as follows:
server1: eth2 -> server2 eth3 -> server3 server2: eth2 -> server3 eth3 -> server1 server3: eth2 -> server1 eth3 -> server2 In other words, a full ring. I want to assign each host an IP, and allow traffic to flow around the ring in the most optimal way. The easiest/most obvious way to do this is via bridges with 'bridge_stp on', but that causes a bottleneck on one host, and one of the 3 P2P links goes unused.
I also tried crafting a fairly complex point-to-point route setup (below), and while this works prefectly while all 3 P2P links are healthy, it doesn't fail over to the second path upon link failure.
How can I accomplish this?
Route setup (server1 as an example): 10.7.0.1/32 is the cluster IP, 10.7.1.1/30 and 10.7.1.10/30 are the P2P links. This setup (sans IPs) is the same on all 3 hosts.
auto lo iface lo inet loopback post-up ip address add 10.7.0.1/32 dev lo post-up sysctl -w net.ipv4.ip_forward=1 auto eth2 iface eth2 inet static address 10.7.1.1 netmask 255.255.255.252 post-up ip route add 10.7.2.2/32 via 10.7.1.1 dev eth2 metric 1 post-up ip route add 10.7.2.3/32 via 10.7.1.2 dev eth2 metric 2 auto eth3 iface eth3 inet static address 10.7.1.10 netmask 255.255.255.252 post-up ip route add 10.7.2.3/32 via 10.7.1.10 dev eth3 metric 1 post-up ip route add 10.7.2.2/32 via 10.7.1.9 dev eth3 metric 2 EDIT: Ended up solving this by implementing OSPF with a few choice static routes, and the cluster address still on a loopback. Aside from the static routes, the interfaces file is as above, and the routing was done in BIRD. Currently it works great, failover time is sub-10 seconds, and performance is perfect in healthy state, while still workable in a failover state (though slightly worse for some links than the STP bridge, and equal for the rest). Still needs a fair amount of testing and tweaking on my part, but here's the BIRD config (as an Ansible template) I used, should anyone be interested: http://pastebin.com/kpmk80aK
[link][13 comments]