VSAN 6.0 Part 4 – All-Flash VSAN Capacity Tier Considerations

In Virtual SAN version 6.0, VMware introduced support for an all-flash VSAN. In other words, both the caching layer and the capacity layer could be made up of flash-based devices such as SSDs.  However, the mechanism for marking some flash devices as being designated for the capacity layer, while leaving other flash devices as designated for the caching layer, is not at all intuitive at first glance. For that reason, I’ve included some steps here on how to do it.

There are a number of ways of achieving this. It can be done on a host-by-host basis, or it could be done centrally via a new Ruby vSphere Console (RVC) command. Let’s take a quick look at one of the ESXi hosts that I wish to have in the VSAN cluster. It has four SSD devices. My wish is to create a single disk group with one of these SSDs as the caching device in the caching tier, while designating the other three SSD devices for capacity.

pre-setup SSDsYou can click on the image above to make it larger. As you can hopefully see, these are all all flash devices. Using the command below, you can see that the first device I am going to work on is an Intel SSD.

# esxcli storage core device list -d naa.55cd2e404b44aa9a
naa.55cd2e404b44aa9a
Display Name: Local ATA Disk (naa.55cd2e404b44aa9a)
Has Settable Display Name: true
Size: 381554
Device Type: Direct-Access
Multipath Plugin: NMP
Devfs Path: /vmfs/devices/disks/naa.55cd2e404b44aa9a
Vendor: ATA
Model: INTEL SSDSC2BA40 
Revision: 0270
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: false
Is Local: true
Is Removable: false
Is SSD: true 
<<truncated>>

In order to use this as a capacity device in an all-flash AF-VSAN, it needs to be marked as a flash capacity device. As mentioned, there are a number of ways to do this. First, lets look at how to do this on a per ESXi host basis. Check if it is already marked for capacity by using the following command:

# vdq -qH -d naa.55cd2e404b44aa9a
DiskResults:
DiskResult[0]:
Name: naa.55cd2e404b44aa9a
VSANUUID:
State: Eligible for use by VSAN
ChecksumSupport: 0
Reason: None
IsSSD?: 1
IsCapacityFlash?: 0
IsPDL?: 0
The IsCapacityFlash is set to 0 above, meaning that it is not yet marked as a flash capacity device. Therefore it cannot be currently used as a capacity tier device in AF-VSAN. Let’s now run the command that tags it as a capacity device from the ESXi host.
# esxcli vsan storage tag add -d naa.55cd2e404b44aa9a -t capacityFlash
 Let’s see if it is now tagged as a flash capacity device.
# vdq -qH -d naa.55cd2e404b44aa9a
DiskResults:
DiskResult[0]:
Name: naa.55cd2e404b44aa9a
VSANUUID:
State: Eligible for use by VSAN
ChecksumSupport: 0
Reason: None
IsSSD?: 1
IsCapacityFlash?: 1
IsPDL?: 0

So now the device is tagged as capacity flash. So what exactly did that tagging operation do? What it did was add an SATP claim rule for that device with an enable_capacity_flash option (see the last line below):

# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL
Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description
-------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ----------------------------
VMW_SATP_LOCAL usb system USB Devices
VMW_SATP_LOCAL ide system IDE Devices
VMW_SATP_LOCAL block system RAID Block Devices
VMW_SATP_LOCAL parallel system Parallel SCSI Devices
VMW_SATP_LOCAL sas system Serial Attached SCSI Devices
VMW_SATP_LOCAL sata system Serial ATA Devices
VMW_SATP_LOCAL unknown system Unknown Devices
VMW_SATP_LOCAL naa.55cd2e404b44aa9a enable_capacity_flash user
#

We can now simply repeat this process on a host by host, flash device by flash device basis and mark all the flash devices needed for capacity.

# vdq -qH -d naa.50015178f35f5e59
DiskResults:
DiskResult[0]:
Name: naa.50015178f35f5e59
VSANUUID:
State: Eligible for use by VSAN
ChecksumSupport: 0
Reason: None
IsSSD?: 1
IsCapacityFlash?: 0
IsPDL?: 0
# esxcli vsan storage tag add -d naa.50015178f35f5e59 -t capacityFlash
# vdq -qH -d naa.50015178f35f5e59
DiskResults:
DiskResult[0]:
Name: naa.50015178f35f5e59
VSANUUID:
State: Eligible for use by VSAN
ChecksumSupport: 0
Reason: None
IsSSD?: 1
IsCapacityFlash?: 1
IsPDL?: 0
# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL
Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description
-------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ----------------------------
VMW_SATP_LOCAL usb system USB Devices
VMW_SATP_LOCAL ide system IDE Devices
VMW_SATP_LOCAL block system RAID Block Devices
VMW_SATP_LOCAL parallel system Parallel SCSI Devices
VMW_SATP_LOCAL sas system Serial Attached SCSI Devices
VMW_SATP_LOCAL sata system Serial ATA Devices
VMW_SATP_LOCAL unknown system Unknown Devices
VMW_SATP_LOCAL naa.55cd2e404b44aa9a enable_capacity_flash user
VMW_SATP_LOCAL naa.50015178f35f5e59 enable_capacity_flash user
[root@localhost:~]

Once the steps have been completed, you can see in the final command output that there is a new rule for the second SSD added to the SATP claim rules in the PSA.

One might ask the question about whether these SATP claim rules can be added to the PSA directly. The answer is yes they can, but the preferred method is to use the tagging command shown above when working on ESXi hosts directly. Working directly on claim rules can be problematic unless you know what you are doing.

One additional way to tag/mark flash devices as capacity tier devices for AF-VSAN is through a Ruby vSphere Console (RVC) command. The command is called vsan.host.claim_disks_differently. It takes an ESXi host and a disk identifier as arguments, along with how the disk should be tagged. In this case, we wish to tag the SSD as a capacity flash device, as shown here (the ‘1’ in the command is a numeric reference to the ESXi host in the previous listing):

> ls
0 10.152.4.9 (host): cpu 2*8*2.00 GHz, memory 274.00 GB
1 10.152.4.10 (host): cpu 2*8*2.00 GHz, memory 274.00 GB
2 10.152.4.11 (host): cpu 2*8*2.00 GHz, memory 274.00 GB
3 10.152.4.12 (host): cpu 2*8*2.00 GHz, memory 274.00 GB
> vsan.host_claim_disks_differently 1 -d n aa.50015178f36363ca \
-c capacity_flash
2015-03-23 12:14:46 +0000: Gathering disk information for host 10.152.4.10
2015-03-23 12:14:48 +0000: Done gathering disk information
Disks on host 10.152.4.10:
Claiming Local ATA Disk (naa.50015178f36363ca) as capacity_flash ...
No existing rule enable_capacity_flash, adding ...
Refreshing state
>

Let’s now check back on the ESXi host once the command has been run, and we can see it marked for capacity, as well as the new claim rule added.

# vdq -qH -d naa.50015178f36363ca
DiskResults:
DiskResult[0]:
Name: naa.50015178f36363ca
VSANUUID:
State: Eligible for use by VSAN
ChecksumSupport: 0
Reason: None
IsSSD?: 1
IsCapacityFlash?: 1
IsPDL?: 0
# esxcli storage nmp satp rule list -s VMW_SATP_LOCAL
Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description
-------------- -------------------- ------ ----- ------ --------- --------------------- ---------- ------------- ----------- ----------- ----------------------------
VMW_SATP_LOCAL usb system USB Devices
VMW_SATP_LOCAL ide system IDE Devices
VMW_SATP_LOCAL block system RAID Block Devices
VMW_SATP_LOCAL parallel system Parallel SCSI Devices
VMW_SATP_LOCAL sas system Serial Attached SCSI Devices
VMW_SATP_LOCAL sata system Serial ATA Devices
VMW_SATP_LOCAL unknown system Unknown Devices
VMW_SATP_LOCAL naa.50015178f36363ca enable_capacity_flash user

The above RVC command would need to be repeated for the other flash devices that you wish to have marked as being designated for the capacity tier.

Once all of the flash devices that you wish to mark as capacity have been tagged, you can go ahead and simply enable the cluster in fully automatic mode if you wish. VSAN will now discover which flash devices are tagged for capacity and setup the disk groups accordingly.

post-setup-viewNow we see a disk group of all flash devices, three of which have been tagged for the capacity tier. You should note that in the disk group, these flash devices are shown as HDD in VSAN 6.0.

So I would agree that this is not the most elegant of implementations. Ideally, all of this should be available at the click of a button, and the UI should be able to tell us that there are indeed flash devices in use in the capacity layer. Rest assured this is all coming down the line.