Do RDMs still rely on LUN ID?

Posted on September 26, 2012 by Cormac — 13 Comments ↓

I had an interesting question the other day about whether Raw Device Mappings (aka RDMs) still had a reliance on the LUN ID, especially when it comes to the vMotion of Virtual Machines which have RDMs attached. I remember some time back that we introduced a concept called Dynamic Name Resolution for RDMs, which meant that we no longer relied on a consistent HBA number or even the path to identify the RDM, but do we still use the LUN ID?

To actually find a reference to the requirement to keep the LUN ID consistent across hosts I had to go back to the Fibre Channel SAN Configuration Guide which we shipped with ESX 4.1. In it, it explicitly states “To use RDMs successfully, a given LUN must be presented with the same LUN ID to every ESX/ESXi host in the cluster.” However this guideline only appeared under the EMC & IBM sections of the guide. And I couldn’t find anything in the 5.x documentation.

To make sure that nothing changed around this in 5.x, I did a bit of investigation.I used my NetApp array to present a LUN to two of my ESXi hosts, but used a different LUN ID for each presentation (ID 40 to one host and ID 50 to another host):

I scanned my SAN, and I could see the LUN on each host, but with different LUN IDs.

The VMFS volume on which my VM was deployed was shared to both hosts. I then proceeded to add the Raw Device Mapping to the Virtual Machine. The VM was on host 1, so the RDM had a LUN ID of 50. Then I looked in the RDM meta data file but there was nothing directly in there which references the LUN ID of the RDM. Although the UUID does look suspiciously like part of an NAA ID (another SCSI identifier mechanism), there is definitely no LUN ID reference.

I next used the vmkfstools -q command to look at the mapping:

# vmkfstools -q WinXP-Lite_1.vmdk
Disk WinXP-Lite_1.vmdk is a Non-passthrough Raw Device Mapping
Maps to: vml.020032000060a98000572d54714e346d63444b44744c554e202020

So the RDM maps to a very long VML number. But what is the VML, and how is it generated? VML which is short for VMware Legacy. This is using a combination of Controller, Target, Channel & Lun information, as well as SCSI id & vendor specific info to identify the LUN. We can parse up the VML as follows:

CTL info – 0200320000 (the 32 here is hex for LUN ID 50 – my RDM)
NAA id – 60a98000572d54714e346d63444b4474
Vendor - 4c554e202020 (HEX -> ASCII converts this to ‘LUN’ on NetApp; it differs from array to array)

So, yes, even though the metadata file itself does not have a LUN id reference, it seems that because we are using the VML as a mapping reference which still a reliance on LUN ID.

I now wanted to see the effect this would have on a vMotion operation, so I tried to migrate my Virtual Machine to the other host which had the RDM presented as LUN ID 40 instead of 50. The vMotion operation failed the compatibility check as follows:

And just for kicks, I searched for that error. First hit was KB 1016210. And inside in this KB (which elaborates on the VML layout), you will find the following statement:

To resolve this issue, LUN presentation should be made consistent for every host participating in a cluster that could run the virtual machine, the raw device mapping metadata file should be consistent with that presentation, and vCenter Server’s cache of this information should be accurate.

I think that’s pretty conclusive, don’t you? To finish, I went back to the array and had it present the LUN to all hosts with a matching LUN ID. I was then able to successfully vMotion the Virtual Machine with an RDM between ESXi hosts.

Bottom line – yes, RDMs still have a reliance on LUN IDs matching across all hosts, even in vSphere 5.x.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Violin Memory & 1 Million IOPs from a single VM

Posted on September 24, 2012 by Cormac — 6 Comments ↓

Another Flash Array vendor that I wanted to meet with at this years VMworld in San Francisco was Violin Memory. For those of you who have been following the keynotes at VMworld 2012, one of the things which will have stood out will have been the 1 million IOPs from a single VM. Now 1 million IOPs isn’t something new. Last year, VMware’s performance team published a paper on how they achieved 1 million IOPs from a single vSphere 5.0 host running six virtual machines. But this year, we achieved 1 million IOPs from a single Virtual Machine. And guess what storage the VM was running on? Yep, a Violin Memory Flash Array.

At Violin’s booth at VMworld San Francisco, I caught up with an old friend and colleague of mine, Vinay Gaonkar, and Violin’s director of marketing, Ashish Gupta. They gave me a good overview of Violin Memory’s new 6×00 arrays, and we discussed the various vSphere integration points.

First I asked about the storage object which Violin surfaces up to the ESXi host & which storage protocols the array supports. They can create multiple LUNs of same capability inside of a given array and export them out to the ESXi host. Violin support Fibre Channel, InfiniBand and iSCSI (over 10GigE).

I then went on to ask about the VAAI primitives that Violin currently support on their arrays. There are none supported at present, but Ashish told me that they plan on supporting all the primitives in the very near future. They have started work on implementing the Write Same and XCopy primitives already, with others following soon.

My next question was about the snapshot and clone technology on the 6×000 series arrays. Is it VM-Centric or Volume-Centric? In other words, does it recognise the VMDK as a storage object or just a file on a LUN? Today it is LUN centric, but as Ashish points out, once Viloin integrates with VMware’s upcoming Virtual Volumes technology, their snapshots will of course become VM-Centric. I asked a similar question around their replication technology, and again Ashish stated that it was LUN orientated presently. Again, Virtual Volumes will make a difference here.

My next question was around BC/DR support, and if there were any plans for an SRM/SRA support with their replication technology. Ashish stated that yes, they plan to support Site Recovery Manager and they are hoping to have a solution in place in H1 2013.

One thing I noticed at the booth is that Violin have a vSphere plugin for Management & Monitoring (above). However this is currently C# so I asked about their plans for a web client plugin. Ashish again said that they expect to have a web client plugin for management and monitoring by H1 2013.

I finally finished with a question around Violin’s plans for a VASA plugin? You guessed it – H1 2013.

Violin Memory are a sponsor of VMworld EMEA and will be in Barcelona on October, 2012. If a Monster VM is what you need, then Violin Memory’s Flash Arrays are definitely worth checking out. 1 million IOPS in a 3U, with no single point of failure because of their fully redundant configuration and vRAID algorithms, is a very compelling solution indeed. They have two kinds of arrays, MLC and SLC. While some of the vSphere integration points are not yet there, its good to hear that there is a concerted effort from Violin to deliver on VAAI, VASA, SRM/SRA & a plugin for the web client.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Nimble Storage 2.0 Architecture Updates

Posted on August 16, 2012 by Cormac — 8 Comments ↓

Regular readers of my VMware Storage Blog will be no stranger to Nimble Storage. I’ve blogged about them on a number of occasions. I first came across them at a user group meeting in the UK & I also wrote an article about them when they certified on VMware’s Rapid Desktop Program for VDI.

Nimble Storage have been in touch with me again to share details about their new 2.0 storage architecture. After a very interesting and informative chat with Wen Yu of Nimble, I’m delighted to be able to share these new enhancements with you, in this first post on my new blog site.

Nimble Storage’s new enhancements can be categorized into two areas. The first of these is a new scale out architecture and the second is further integration with vSphere.

Scale to Fit

Scale to Fit architecture is how Nimble Storage describe their new elastic scaling feature. It basically allows customers to scale out their storage on a particular dimension, be it capacity or performance. This new architecture allows customers to start with a small footprint, and then to scale performance and capacity. This can be done without having to migrate any data and without any Virtual Machine/application downtime. The great advantage of this of course is that it avoids over-provisioning of storage up front, keeping initial costs down. When additional performance or capacity is needed, customers only need to grow on that dimension. This means that customers don’t pay for additional performance if they only need capacity, and vice-versa.

vSphere Integration Features

There are 3 new vSphere integration features to call out in this new release.

Nimble Storage have a new Storage Replication Adapter (SRA) for integrating with VMware Site Recovery Manager (SRM). Business Continuance and Disaster Recovery are essential features for any enterprise class storage array, and it is great to see that Nimble now offer full integration with VMware’s BC/DR flagship product.
There are a number of additional VAAI offload primitives supported. The first of these is Hardware Assisted Locking (ATS) which enables ESXi hosts to offload VMFS volume locks to the Nimble storage array. The second is the UNMAP primitive, which enables VMFS volumes built on thin provisioned disks to do space reclamation after storage vMotion or VM deletion. If I remember correctly from previous conversations with Nimble, they already support the WRITE_SAME primitive.
This last feature is the one I am most excited about. Nimble Storage now offer their own Path Selection Plugin (PSP) into the Pluggable Storage Architecture of the VMkernel. This optimized multipathing plugin will load balance I/O, and provide linear performance scalability with a single Nimble storage array or multiple storage arrays in a scale-out cluster. The PSP is called Nimble_PSP_Directed.