Heads Up! VAAI UNMAP issues on EMC VMAX

by Posted on

We just got notification about a potential issue with the VAAI UNMAP primitive when used on EMC VMAX storage systems with Enginuity version 5876.159.102 or later. It seems that during an ESXi reboot, or during a device ATTACH operation, the ESXi may report corruption. The following is an overview of the details found in EMC KB 184320. Other symptoms include vCenter operations on virtual machines fail to complete and the following errors might be found in the VMkernel logs:

WARNING: Res3: 6131: Invalid clusterNum: expected 2624, read 0 
[type 1] Invalid totalResources 0 (cluster 0).[type 1] Invalid  
nextFreeIdx 0 (cluster 0).
WARNING: Res3: 3155: Volume aaaaaaaa-bbbbbbbb-cccc-dddddddddddd 
("datastore1") might be damaged on the disk. Resource cluster  
metadata corruption has been detected

The VAAI UNMAP primitive is used for reclaiming ‘dead’ or ‘stranded’ space on thin provisioned VMFS datastores. It seems the issue is related to a combination of the VAAI Thin Provisioning Stun primitive and the VAAI UNMAP primitive.

To allow a datastore to return useful SCSI sense data when a device on the storage system cannot allocate any more space, a “NULL UNMAP” command is sent to the storage system from the ESXi. This allows ESXi to trigger a Thin Provision Stun on a VM when it requests additional space on an already full datastore. This command, unlike a normal UNMAP, does not UNMAP any blocks.

A normal UNMAP command however, when sent from the ESXi to the storage system, populates a buffer in the storage system with UNMAP descriptor information. What EMC has found is that the “NULL UNMAP” used for TP Stun does not send a descriptor with the command so the buffer remains uninitialized. This is the root of the issue.

When the VMAX storage system processes the UNMAP commands, the “NULL UNMAP” commands are also processed but since the descriptor information hasn’t initialized the buffer, the command uses residual information found in the internal buffer. This causes blocks which could still be in use to become unmapped.

EMC already have a fix which can be requested through the EMC Support Center. EMC  support will verify if your VMAX is exposed to this Enginuity issue and a fix (Fix 72255) has already been written to address the problem. I’ve been informed that this can be requested through the Enginuity Pack (epack) process.

Of course, if you are affected, it is advisable not to use UNMAP (via vmkfstools -y or the new esxcli namespace) until the fix is in place.

Published by Cormac

11 Replies to “Heads Up! VAAI UNMAP issues on EMC VMAX”

  1. Are there any situations where an UNMAP command might occur automatically or is this strictly a manual operation via vmkfstools or esxcli? Our Storage Engineers are trying to determine the timing and risk. Thanks!

    1. If I remember correctly, only the vanilla 5.0 release of ESXi/vSphere had it automated. Since then, its either been completely disabled or it has been a manual process. You may also disable it completely if you wish using the Advanced Setting /VMFS3/EnableBlockDelete. VMware KB article 2007427 has more detail around this.

    2. Just got an update on this – ideally you should disable UNMAP @ the array layer. If there is worry in the mean time, avoid net-new disk detection by ESXi (Rebooting hosts / presenting new storage).

      1. Cormac,

        So even if it is disabled at the ESXi layer with the advanced setting we are still at risk?

        1. It would appear so – the directive is to not reboot and/or add any disks to the hosts. It seems that the NULL UNMAP used for Thin Provisioned datastores is the crux of the issue and not the actual dead space reclaim UNMAPs.

  2. When I read the EMC KB earlier this week, I read that it this only happens if you reboot or add storage during a deadspace reclaim operation. As long as you are not doing deadspace reclamation a host reboot or storage add should not pose an issue, correct?

    1. Correct.
      Unless you “patched” your Array, don’t:
      – Reboot a host
      – Rescan for VMFS devices
      – Have the Array provision new SCSI devices to the host (it will ninja rescan)
      – Actually issue unmap commands to the array….

  3. Pingback: » Heads Up Alert: vSphere 5.5 U1 NFS Random Disconnection Bug! Long White Virtual Clouds
  4. Pingback: » Heads Up Alert: vSphere 5.5 U1 NFS Random Disconnection Bug! Long White Virtual Clouds

  5. Esx 5.1 and 5.6 will send an unmap to the array when booting to check if the array can support it. Reboot or vmfs rescan = lost data.

Comments are closed.