Hi, I'm pretty new to GlusterFS and I'm having what feels like a newbie problem but I can't find a fix. I have a 3 node GFS set. Each node has 1 brick and I have a single volume that has replication set to three.
On the client side I mount using glusterfs and I connect to a dns name (gluster.stuff.int) that round robin resolved to the 3 gluster nodes.
So what I think I should have is a loadbalanced system where any of my gluster nodes should be able to service a request. Since my clients should all be making an initial connection to a different node (because of the DNS round robin) I also think network load should be shared across the entire cluster.
wrong!
What I have instead is what appears to be an active/passive type cluster. My gfs01 node gets all the network traffic. The other 2 node are almost completely idle. Under load I'll see gfs01 pushing over 500Mb/s while the other 2 nodes are under 1Mb/s.
On this system HA appears to work fine. If I reboot gfs01 mid-job, one of the other 2 nodes will immediately spring to life and start handing the requests. But as soon as GFS01 completes his reboot he assumes the load.
This feels like a feature, it feels like something some people would want in certain situations. But it's not what I want - I want my network to be fully load balanced.
Anybody have any ideas what could be going on? Thanks in advance for any help!
*** Edit *** I should probably mention that my use case is also a little weird. This isn't a general File Share where you might expect random access to a large number of different files. I'm using it as a file store for a another cluster of servers that all work on the same task and need the same files. So my compute cluster will all need the same 2GB of files and will all in unison pull from GFS to a local cache. So that means all 70 servers want the same files all at once.
It feels like maybe there is some file caching in GFS and that maybe 1 node "owns" a file in the cache and that's why that one node is shouldering the entire load. I can find things in the doc that imply that, but not that say that outright.
Thanks again!
[link] [comments]