Just thought I’d bring to your attention something that has been doing the rounds here at VMware recently, and will be applicable to those of you using QLogic HBAs with ESXi 5.x. The following are the device queue depths you will find when using QLogic HBAs for SAN connectivity:
- ESXi 4.1 U2 – 32
- ESXi 5.0 GA – 64
- ESXi 5.0 U1 – 64
- ESXi 5.1 GA – 64
The higher depth of 64 has been this way since 24 Aug 2011 (the 5.0 GA release). The issue is that this has not been documented anywhere. For the majority of users, this is not an area of concern and is probably a benefit. But there are some concerns.
But for those of you with a significantly large number of LUNs presented to an ESXi host with a QLogic HBA, this may be a consideration. Each device will now consume 64 slots rather than 32 slots in the adapter queue (which for QLogic is 4096). For instance, a queue depth of 32 will allow you to have 128 LUNs (32 x 128 = 4096). Now with this raised to 64, this reduces the number of LUNs that may have this full queue depth value to 64 (64 x 64 = 4096). If you hit the adapter queue limit, then you won’t be able to reach the device queue depth, and may possibly have I/Os retried due to queue full conditions. You can use esxtop to monitor queue depth usage under disk statistics.
Why did we change it? The reason was for Storage I/O Control (SIOC) improvements. It was increased to handle situations when there were a large number of VMs per datastore and to give SIOC a queue depth of 64 to play with rather than 32 for VM performance/fairness. If the VMs on one host are more active in terms of I/O activity and need a larger share of the I/O bandwidth to the same device/LUN, this gives SIOC the ability to do more for these VMs.
The bottom line is that this is a significant change between 4.x and 5.x (and something that I wasn’t aware that we had even changed). It is something to keep in mind if you are planning to upgrade to 5.x, or if you have already moved to 5.x and you use QLogic HBAs for SAN connectivity with a lot of LUNs/devices. Emulex device queue depth settings have not changed between releases. It is still 30 if I remember correctly.