In Virtual SAN version 6.0, VMware introduced support for an all-flash VSAN. In other words, both the caching layer and the capacity layer could be made up of flash-based devices such as SSDs. However, the mechanism for marking some flash devices as being designated for the capacity layer, while leaving other flash devices as designated for the caching layer, is not at all intuitive at first glance. For that reason, I’ve included some steps here on how to do it.
There are a number of ways of achieving this. It can be done on a host-by-host basis, or it could be done centrally via a new Ruby vSphere Console (RVC) command. Let’s take a quick look at one of the ESXi hosts that I wish to have in the VSAN cluster. It has four SSD devices. My wish is to create a single disk group with one of these SSDs as the caching device in the caching tier, while designating the other three SSD devices for capacity.
# esxcli storage core device list -d naa.55cd2e404b44aa9a naa.55cd2e404b44aa9a Display Name: Local ATA Disk (naa.55cd2e404b44aa9a) Has Settable Display Name: true Size: 381554 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.55cd2e404b44aa9a Vendor: ATA Model: INTEL SSDSC2BA40 Revision: 0270 SCSI Level: 6 Is Pseudo: false Status: on Is RDM Capable: false Is Local: true Is Removable: false Is SSD: true <<truncated>>
In order to use this as a capacity device in an all-flash AF-VSAN, it needs to be marked as a flash capacity device. As mentioned, there are a number of ways to do this. First, lets look at how to do this on a per ESXi host basis. Check if it is already marked for capacity by using the following command:
# vdq -qH -d naa.55cd2e404b44aa9a DiskResults: DiskResult[0]: Name: naa.55cd2e404b44aa9a VSANUUID: State: Eligible for use by VSAN ChecksumSupport: 0 Reason: None IsSSD?: 1 IsCapacityFlash?: 0 IsPDL?: 0
# esxcli vsan storage tag add -d naa.55cd2e404b44aa9a -t capacityFlash
# vdq -qH -d naa.55cd2e404b44aa9a DiskResults: DiskResult[0]: Name: naa.55cd2e404b44aa9a VSANUUID: State: Eligible for use by VSAN ChecksumSupport: 0 Reason: None IsSSD?: 1 IsCapacityFlash?: 1 IsPDL?: 0
So now the device is tagged as capacity flash. So what exactly did that tagging operation do? What it did was add an SATP claim rule for that device with an enable_capacity_flash option (see the last line below):
# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ---------------------------- VMW_SATP_LOCAL usb system USB Devices VMW_SATP_LOCAL ide system IDE Devices VMW_SATP_LOCAL block system RAID Block Devices VMW_SATP_LOCAL parallel system Parallel SCSI Devices VMW_SATP_LOCAL sas system Serial Attached SCSI Devices VMW_SATP_LOCAL sata system Serial ATA Devices VMW_SATP_LOCAL unknown system Unknown Devices VMW_SATP_LOCAL naa.55cd2e404b44aa9a enable_capacity_flash user #
We can now simply repeat this process on a host by host, flash device by flash device basis and mark all the flash devices needed for capacity.
# vdq -qH -d naa.50015178f35f5e59 DiskResults: DiskResult[0]: Name: naa.50015178f35f5e59 VSANUUID: State: Eligible for use by VSAN ChecksumSupport: 0 Reason: None IsSSD?: 1 IsCapacityFlash?: 0 IsPDL?: 0
# esxcli vsan storage tag add -d naa.50015178f35f5e59 -t capacityFlash
# vdq -qH -d naa.50015178f35f5e59 DiskResults: DiskResult[0]: Name: naa.50015178f35f5e59 VSANUUID: State: Eligible for use by VSAN ChecksumSupport: 0 Reason: None IsSSD?: 1 IsCapacityFlash?: 1 IsPDL?: 0
# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ---------------------------- VMW_SATP_LOCAL usb system USB Devices VMW_SATP_LOCAL ide system IDE Devices VMW_SATP_LOCAL block system RAID Block Devices VMW_SATP_LOCAL parallel system Parallel SCSI Devices VMW_SATP_LOCAL sas system Serial Attached SCSI Devices VMW_SATP_LOCAL sata system Serial ATA Devices VMW_SATP_LOCAL unknown system Unknown Devices VMW_SATP_LOCAL naa.55cd2e404b44aa9a enable_capacity_flash user VMW_SATP_LOCAL naa.50015178f35f5e59 enable_capacity_flash user [root@localhost:~]
Once the steps have been completed, you can see in the final command output that there is a new rule for the second SSD added to the SATP claim rules in the PSA.
One might ask the question about whether these SATP claim rules can be added to the PSA directly. The answer is yes they can, but the preferred method is to use the tagging command shown above when working on ESXi hosts directly. Working directly on claim rules can be problematic unless you know what you are doing.
One additional way to tag/mark flash devices as capacity tier devices for AF-VSAN is through a Ruby vSphere Console (RVC) command. The command is called vsan.host.claim_disks_differently. It takes an ESXi host and a disk identifier as arguments, along with how the disk should be tagged. In this case, we wish to tag the SSD as a capacity flash device, as shown here (the ‘1’ in the command is a numeric reference to the ESXi host in the previous listing):
> ls 0 10.152.4.9 (host): cpu 2*8*2.00 GHz, memory 274.00 GB 1 10.152.4.10 (host): cpu 2*8*2.00 GHz, memory 274.00 GB 2 10.152.4.11 (host): cpu 2*8*2.00 GHz, memory 274.00 GB 3 10.152.4.12 (host): cpu 2*8*2.00 GHz, memory 274.00 GB
> vsan.host_claim_disks_differently 1 -d n aa.50015178f36363ca \ -c capacity_flash 2015-03-23 12:14:46 +0000: Gathering disk information for host 10.152.4.10 2015-03-23 12:14:48 +0000: Done gathering disk information Disks on host 10.152.4.10: Claiming Local ATA Disk (naa.50015178f36363ca) as capacity_flash ... No existing rule enable_capacity_flash, adding ... Refreshing state >
Let’s now check back on the ESXi host once the command has been run, and we can see it marked for capacity, as well as the new claim rule added.
# vdq -qH -d naa.50015178f36363ca DiskResults: DiskResult[0]: Name: naa.50015178f36363ca VSANUUID: State: Eligible for use by VSAN ChecksumSupport: 0 Reason: None IsSSD?: 1 IsCapacityFlash?: 1 IsPDL?: 0
# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description -------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ---------------------------- VMW_SATP_LOCAL usb system USB Devices VMW_SATP_LOCAL ide system IDE Devices VMW_SATP_LOCAL block system RAID Block Devices VMW_SATP_LOCAL parallel system Parallel SCSI Devices VMW_SATP_LOCAL sas system Serial Attached SCSI Devices VMW_SATP_LOCAL sata system Serial ATA Devices VMW_SATP_LOCAL unknown system Unknown Devices VMW_SATP_LOCAL naa.50015178f36363ca enable_capacity_flash user
The above RVC command would need to be repeated for the other flash devices that you wish to have marked as being designated for the capacity tier.
Once all of the flash devices that you wish to mark as capacity have been tagged, you can go ahead and simply enable the cluster in fully automatic mode if you wish. VSAN will now discover which flash devices are tagged for capacity and setup the disk groups accordingly.
So I would agree that this is not the most elegant of implementations. Ideally, all of this should be available at the click of a button, and the UI should be able to tell us that there are indeed flash devices in use in the capacity layer. Rest assured this is all coming down the line.