SAS Expander support on Virtual SAN

This question has come up on a number of occasions in the past. It usually comes up when there is a question about scalability and the number of disk drives that can be supported on a single host that is participating in Virtual SAN. The configurations maximums for a Virtual SAN node is 5 disk groups, with each disk group containing 1 flash device and up to 7 capacity devices (these capacity devices are magnetic disks in hybrid configurations or flash devices in all-flash configurations). Now the inevitable next question is how is this configuration implemented on a physical server. How can I get to 35/40 devices in a single server. There are a few ways to do it.

Multiple storage controllers

This is probably the simplest approach. Most storage controllers will support either 8 or 16 disk drives. If you have a controller that supports 8 disk drives, then you’re looking at 5 controllers to reach the maximum number of physical disks per VSAN hosts. If you have a controller that supports 16 disks drives, the you would need 3 controllers. This configuration isn’t always possible, simply due to the number of available PCIe slots for controllers.

SAS expanders

SAS expanders are yet another option. Let’s describe a SAS expander first. Typically it is a PCIe device that plugs into the PCIe bus of the server, and is connected to an external JBOD (Just a Bunch of Disk) or external storage controllers which contains a number of storage devices. Communication travels from the storage controller thru the SAS expander to the disk device.

We’ve had to do some work at VMware to qualify the SAS expanders as we didn’t know if they would be suitable for Virtual SAN Environments. We put this note in our VSAN 6.0 Design and Sizing Guide:

“SAS expanders are sometimes considered to extend the number of storage devices that can be configured with a single storage I/O controller. VMware has not extensively tested SAS expanders with VSAN, and thus does not encourage their use. In addition to potential compatibility issues, the use of SAS expanders may impact performance and increase the impact of a failed disk group.”

Well, that was a number of months ago, and now we have done the first certification of these devices. The DELL R730XD is now fully certified with SAS expanders. The R730XD has “built-in” SAS expanders (it’s not an optional add on).  This means that you can populate 24 drives behind a controller without any additional hardware.

However it is important to remember that having a fully populated SAS expander configuration with all drives behind a single controller may not provide as good a performance when compared to using multiple controllers. Configurations of this nature can be considered for workloads that are capacity intensive and may not require the highest level of performance.  If performance is an important factor, then it may be worth considering adding additional controllers. Again, this is dependent on the requirements of the workload.

Designing & sizing a VSAN deployment

When designing a VSAN deployment, you must consider the capacity requirement. Does it require more capacity per host than a single controller can handle? If so, what options are available; multiple controllers or SAS expander – and more importantly, is optimal capacity more important than optimal performance? If SAS expander is the choice your are going with, make sure the host in question is qualified.

[Update] I was asked where I can find the list of SAS expanders that are currently supported. At present, SAS expanders are only supported with select VSAN ready nodes. You can get a list of VSAN ready nodes, and thus which SAS expanders are supported from clicking on this link.