Bear with me a minute. Is there any way to force Linux (I'm using CentOS 6.2, fwiw) to not use local IP routing when connecting to an IP address hosted on the local box?
Here's my setup:
- Server-A at 192.168.56.101
- Server-B at 192.168.56.102
- Floating database IP is at 192.168.56.103. The database runs on either Server-A or Server-B. The applications that use the database are running active-active on both servers simultaneously.
I'm using clustering software (VCS) that remounts the storage from Server-A to Server-B, brings the IP from Server-A to Server-B, then will restart the database. This helps when there are situations where the database crashes, or where we want to transfer the database off a system to do Linux patching (and reboot).
There is one problem - when the database fails over, the application where the database was local previously crashes. The application on the remote server recovers just fine from brief database unavailability.
But it gets worse - if I bring the application back up, and failover in the reverse direction, I see the same issue - whichever server the database vacates, that local application instance crashes. So it goes both ways.
From what I can determine, this is because the application begins to use the database's IP address for all it's outbound communication! Thus when the database fails over, that application loses ability to access all of its connections that it had open.
Here's how I would illustrate the issue:
From Server-A (database running here at the moment):
$ ip route get 192.168.56.101 local 192.168.56.101 dev lo src 192.168.56.101 cache <local> mtu 16436 advmss 16396 hoplimit 64 $ ip route get 192.168.56.102 192.168.56.102 dev eth0 src 192.168.56.101 cache mtu 1500 advmss 1460 hoplimit 64 $ ip route get 192.168.56.103 local 192.168.56.103 dev lo src 192.168.56.103 cache <local> mtu 16436 advmss 16396 hoplimit 64 From Server-B:
$ ip route get 192.168.56.101 192.168.56.101 dev eth0 src 192.168.56.102 cache mtu 1500 advmss 1460 hoplimit 64 $ ip route get 192.168.56.102 local 192.168.56.102 dev lo src 192.168.56.102 cache <local> mtu 16436 advmss 16396 hoplimit 64 $ ip route get 192.168.56.102 192.168.56.103 dev eth0 src 192.168.56.102 cache mtu 1500 advmss 1460 hoplimit 64 Is there any way to force the database IP, 192.168.56.103, to go out through the primary physical IP, even if 192.168.56.103 is local? That way the application querying the database won't start using the database's own IP for those queries. Or any other thoughts on how to deal with this issue? I might be thinking about this all wrong. Thanks for the advice!
(Edit: formatting)
[link][5 comments]