(If this belongs somewhere else, I apologize up front, just point me in the right direction)
I've started a little side project at work and was looking for input on anyone who's been down this road. Currently VMWare with HP servers backed by a 3Par F400 (10 and 15k FC) and Brocade 300 4Gb FC. Considering how absurd the pricing is and how badly HP keeps trying to screw me, I'm looking at building my own SAN.
We setup a small test, a pair of DL360 G6 with 10Gb Intel X540-T1 cross connected. CentOS 6 on one exporting an iscsi target composed of 4 SAS disks in raid0 with ietd and the other box running vSphere. We were able to get ~5.5Gb/s off four disks, so it's looking promising so far. I'm thinking with whitebox servers loaded with eMLC SSD and/or some combination of SAS and maybe even some SATA mixed in, I could build a pretty high performance SAN. Add in a relatively inexpensive 10Gb switching infrastructure (Arista, Quanta, Force10, open to ideas here) and I should be good to go. We're talking about a dozen physical VMWare hosts and ~20TB. eMLC SSD is running from $1-$2/GB depending on where you buy (Mushkin < Seagate < Intel, least to most expensive, in that order) and 15k SAS is around $0.50-$1.00/GB. Considering I spent somewhere over $20/GB when I originally deployed our 3Par SAN (ouch) and it's full of 10-15k spinners, I feel like I could build something that would outperform it for a fraction of the cost.
Would love any input, especially as it relates to filesystem selection. I'd love to use ZFS but not sure about Linux support which makes me want to take a hard look at btrfs. I'd of course move to full blown Redhat for some level of support and I'm totally open on the iscsi target software, no preference at all. Anyone know how much support I could get from Redhat if I was using a scsi target from the official repos? What about something really crazy like ceph?
Anyone been down this road before? Anyone have a write up on any custom builds? Am I totally batshit insane?
If I can get some feedback I'll post the hardware configurations I've been considering. If nothing else it's an interesting thought exercise.
edit from feed back (thanks everyone so far!):
- Yes, support is definitely a concern, I totally agree
- NFS may be a much better option for me, we're going to run some tests
[link][84 comments]