Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).
Could not truncate file .vmfsBalloonrYq2EH to 45.4 TB (File too large).

Eventually the UNMAP process will reduce the amount of space that we try to reclaim, and will get down to the 2TB threshold and start working. However this can take some time, especially if you are trying to reclaim a large amount of dead space. Therefore caution must be used when running the vmkfstools -y “%” command. The % value supplied as an argument to this command represents the % of free space to reclaim as dead space. We advise you to work out how much free space is on the volume, and then ensure that the % value is set so as not to represent dead space with a value greater than 2TB.

For example, if the free space was shown to be 30TB and you thought that there was 3TB worth of dead space to reclaim on the volume, setting the percentage value to 10% in vmkfstools would result in a 3TB balloon file. This exceeds the 2TB limit and may cause the issue described above. Therefore it would be best to run the command twice, each time with a 5% setting. This way only 1.5TB of space is reclaimed at any one time, and the temporary balloon file is limited to 1.5TB in size.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Autocache, is an I/O acceleration product which uses flash devices. It is installed on the hypervizor, inspects the I/O from all virtual machines and caches the hot I/O onto a local PCIe flash card or solid-state disk (SSD). It is a read accelerator, which means that writes go directly to persistent storage (write-thru), but those writes are cached so that future read operations are serviced from flash. Of course this is also beneficial for writes, since if the reads are serviced by flash, the spinning disks are not accessed for reads, thus more writes. Autocache works for virtual machines on both block and NFS.

Since it is installed on the hypervizor, Autocache does not install any agents in the guest OS, It supports vSphere HA  and core vSphere features such as vMotion. It is integrated into the existing workflows in the VMware vSphere client and is managed by an Autocache tab in the client. Here is an example of a statistics view from the Autocache tab in the vSphere client:

As mentioned, Autocache supports vMotion. When a vMotion is detected, Autocache pre-warms cached data from the source host to the destination host, thereby giving vMotion a head start on making critical data available faster.

The fact that this is an agent-less product, fully integrated with core vSphere features like HA & DRS, supports vMotion and has negligible overhead on the hypervizor makes this a very interesting product. Proximal data claim negligible CPU overhead and memory consumption of about 1/1400th of flash capacity. Proximal Data tell me that customers can typically expect a 2x-3x in virtual machine density, depending on workload.

Currently, Autocache supports VMware ESXi 4.1 and 5.0/5.1.

I’ll admit that I haven’t had the opportunity to try out the product – this time of year is always rather busy for us in VMware Technical Marketing. The content here is based on conversations with Rich over the last couple of months. However, Proximal Data allow you to download and evaluate the product, so I’d say give this a go for yourselves. The download is available here. If you do get a chance to evaluate, I’d love to hear your thoughts so please leave a comment.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Many of you will be aware of VMware’s recent introduction of the SE Sparse Disk. This is VMware’s first effort to allow you to reclaim dead space from within the Guest OS. Right now, it is only supported with VMware Horizon View version 5.2, so basically is limited to View desktops. For general purpose virtual machines, we still do not have a solution.

We had some workarounds in the past – notably running SDelete to do a zero fill operation within the (Windows) Guest OS, and then migrate the VM between datastores of differing block sizes. Well, that method was lost with the introduction of a unified 1MB file block size with VMFS-5. So I was very interested when Bob reached out about their new PerfectStorage solution for reclaiming dead space within a Guest OS.

In Bob’s own words: “PerfectStorage includes an enterprise console that allows the product to be deployed, installed and scheduled across the physical and virtual servers in the enterprise. PerfectStorage can be scheduled to run unattended,  on multiple systems at the convenience of the system/storage administrator and there is no need to shut down system activity.   The product can perform an analysis of each thin-provisioned volume and provide an estimate of how much space can be reclaimed.  The SMART Zero-Fill feature will only write zeros to those clusters that are not already zeroed.  Administrators opt for the type of reclamation they want to do and PerfectStorage performs the Zero-Fill or issues the UNMAP command for the reclaimable space.

The beta testers who had previously used SDelete-like tools said PerfectStorage saved them hours of time working with the command line and baby-sitting the reclamation process. Larger users said PerfectStorage allowed them to reclaim significant space which helped them better plan future storage purchases.

 PerfectStorage will do Zero-Fill on any storage controller that has zero detection capability for Windows Vista and above and Windows 2003 SP1 and above.  To perform the UNMAP function requires VMware 5.1 and Virtual Hardware 9 on the Windows guests or Hyper-V 2008/R2 or 2012″

PerfectStorage product trials are now available at http://download.raxco.com/perfectstorage.  The link has two places where it asks “Find Out if PerfectStorage is Right For You”.  Based on some questions that arose during early beta, Raxco put together a short decision tree that lets an administrator know if PerfectStorage will work for them.

Seems like this could be a nice tool, and I know this is an area of pain for many administrators. Worth checking out in my opinion.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

• ESXi 4.1 U2 – 32
• ESXi 5.0 GA – 64
• ESXi 5.0 U1 – 64
• ESXi 5.1 GA – 64

The higher depth of 64 has been this way since 24 Aug 2011 (the 5.0 GA release). The issue is that this has not been documented anywhere. For the majority of users, this is not an area of concern and is probably a benefit. But there are some concerns.

But for those of you with a significantly large number of LUNs presented to an ESXi host with a QLogic HBA, this may be a consideration. Each device will now consume 64 slots rather than 32 slots in the adapter queue (which for QLogic is 4096). For instance, a queue depth of 32 will allow you to have 128 LUNs (32 x 128 = 4096). Now with this raised to 64, this reduces the number of LUNs that may have this full queue depth value to 64 (64 x 64 = 4096). If you hit the adapter queue limit, then you won’t be able to reach the device queue depth, and may possibly have I/Os retried due to queue full conditions. You can use esxtop to monitor queue depth usage under disk statistics.

Why did we change it? The reason was for Storage I/O Control (SIOC) improvements. It was increased to handle situations when there were a large number of VMs per datastore and to give SIOC a queue depth of 64 to play with rather than 32 for VM performance/fairness.  If the VMs on one host are more active in terms of I/O activity and need a larger share of the I/O bandwidth to the same device/LUN, this gives SIOC the ability to do more for these VMs.

The bottom line is that this is a significant change between 4.x and 5.x (and something that I wasn’t aware that we had even changed). It is something to keep in mind if you are planning to upgrade to 5.x, or if you have already moved to 5.x and you use QLogic HBAs for SAN connectivity with a lot of LUNs/devices. Emulex device queue depth settings have not changed between releases. It is still 30 iirc.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

Many folks might ask why this replication feature is so special. Well, from the offset, Tintri were always about doing things are the per-VM level. Almost all array level replication technologies continue to work at the per-LUN or per-Volume basis, meaning that you are replicating a lot of unnecessary blocks over the wire. Tintri’s replication feature is only replicating the blocks used by the virtual machine that you wish to protect.

Secondly, with Tintri’s replication feature, if a block already exists on the destination array, then they will not copy that block over the wire. Instead, they simply reference it via metadata. All of Tintri’s replication is deduplicated and compressed, meaning that the minimum amount of data is sent over the wire.

Another key feature of this replication technology is a feature which Tintri refer to as Remote Cloning. This allows administrators to clone a virtual machine from a local Tintri storage array to a remote Tintri array at another site. This is very interesting, and I’m sure this is something many administrators wish they could do with the click of a button from time to time. One nice part of this remote cloning feature is that the thin provisioned nature of the virtual machine is preserved, i.e. when the VM is cloned to the destination, it is not inflated to a full size, but instead remains thin. The VM may also be cloned back to the source from the destination. Tintri feel that this ability to retain thinness and do bidirectional cloning across arrays gives them an advantage over their competition. This was a very interesting part of the demo at VMworld 2012, where Tintri were able to show hundreds of virtual machines being cloned to a remote array in a matter of seconds.

The 2.0 release should be available very soon. Looks like it has some nice new functionality for existing Tintri customers, and definitely something to look into for folks looking for arrays that work at a virtual machine granularity. One minor disappointment for me is that the new replication feature is not yet integrated with VMware’s Site Recovery Manager product, and I hope this is something Tintri choose to implement in the not too distant future.

Update: Here’s the press release: http://www.tintri.com/news/press-releases/tintri-introduces-next-generation-vm-aware-storage-functionality-software. Looks like it will be shipping in mid-May, 2013.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

This new version now supports Android devices as well as IOS iPad device.

Please check out the post by Alan Renouf on the vSphere blog to learn more, see some sample content and the links on where to download the app.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

In a nutshell, this all related to how the VMDK behaves when a virtual machine snapshot is taken. VMDKs, by default, are said to be dependent. In other words, when the virtual machine is snapshot’ed, this disk is included in the snapshot. Independent disks, however, can have different behaviors when the VM is snapshot’ed. Let’s examine the difference next.

Independent Persistent Mode

When a VMDK is configured in Independent Persistent Mode, what you will see is that no delta file is associated with this disk during a snapshot operation. In other words, during a snapshot operation, this VMDK continues to behave as if there is no snapshot being taken of the virtual machine and all writes go directly to disk. So there is no delta file created when a snapshot of the VM is taken, but all changes to the disk are preserved when the snapshot is deleted.

Independent Non-persistent Mode

When a VMDK is configured as Independent Non-persistent Mode, a redo log is created to capture all subsequent writes to that disk. However, if the snapshot is deleted, or the virtual machine is powered off, the changes captured in that redo log are discarded for that Independent Non-persistent VMDK.

Why Independent Disks?

This begs the question of what do Independent disks do for you, be they persistent or non-persistent. The bottom line is that if a disk is Independent, they can be omitted from a backup operation by virtue of the fact that they do not support snapshot operations. You can think of Independent as meaning ‘Independent of snapshots’.

Sample Use Case

Let’s take a virtual machine which hosts a web service, but it is primarily a read-only web site with content that rarely changes. This VM has three disks. Disk 1 contains the Guest OS and web application (e.g. Apache). Disk 2 contains the web pages for the web site. Disk 3 contains all the logging activity.

In this case, disk 1 (OS & app) are dependent (default) settings and is backed up nightly. Disk 2 is independent non-persistent (not backed up, and any changes to these pages will be discarded). Disk 3 is independent persistent (not backed up, but any changes are persisted to the disk).

If updates are needed to the web site’s pages, disk 2 must be taken out of independent non-persistent mode temporarily to allow the changes to be made.

Now lets say that this site gets hacked, and the pages are doctored with something which is not very nice. A simple reboot of this host will discard the changes made to the web pages on disk 2, but will persist the logs on disk 3 so that a root cause analysis can be carried out.

There you have it. Hopefully this gives you an idea of the reasoning behind independent persistent and independent non-persistent disks.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

In a nutshell, ESXi has a limited amount of VMFS heap space by default. While you can increase it from the default to the maximum, there are still some gaps. When you create very many VMDKs on a very large VMFS volume, the double indirect pointer mechanism to address the blocks way out in the address space consume heap. The result is that although we supported very large VMFS volumes (up to 64TB), the reality up to now is that a single host (since heap is defined on a per host basis) could only address in the region of 30TB of open files. This isn’t always an issue, since typically VMFS is a clustered file system and is shared by many hosts. Therefore one would typically have the open VMDKs spread across many hosts in a cluster. However it is an issue for stand-alone hosts with lots of virtual machines with lots of VMDKs, and is also an issue for hosts which want to have a virtual machine with a lot of VMDKs attached, for the purposes of a file share for example.

Anyway, to cut to the chase, a recent patch release for ESXi 5.0 increases the default heap size to 256MB and maximum heap size to 640MB per ESXi host. This should allow a single ESXi host to access in the region of 60TB open VMDK. Previously the default was 80MB and the maximum was 256MB, so we have increased this significantly. This is pretty much the maximum size of the VMFS volume anyway. The patch is Patch ESXi500-201303401-BG.

Although the patch for ESXi 5.1 is not yet out, it should be available very shortly, and will have a similar fix.

For those of you using very large VMFS volume with lots of virtual machines disk files, consider scheduling a maintenance slot very soon to apply these patches. This is not an issue for NFS, fyi.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

]]> http://cormachogan.com/2013/04/11/heads-up-new-patches-for-vmfs-heap/feed/ 2 http://cormachogan.com/2013/04/08/microsoft-clustering-on-vsphere-incompatible-device-errors/ http://cormachogan.com/2013/04/08/microsoft-clustering-on-vsphere-incompatible-device-errors/#comments Mon, 08 Apr 2013 14:31:09 +0000 Cormac http://cormachogan.com/?p=1679 Continue reading →]]> When setting up a Microsoft Cluster with nodes running in vSphere Virtual Machines across ESXi hosts, I have come across folks who have experienced Incompatible device backing specified for device ‘0’ errors. These are typically a result of the RDM (Raw Device Mapping) setup not being quite right. There can be a couple of reasons for this, as highlighted here.

Different SCSI Controller

On one occasion, the RDM was mapped to the same SCSI controller as the Guest OS boot disk. Once the RDM was moved to its own unique SCSI controller, it resolved the issue. Basically, if the OS disk is configured to use SCSI 0:0, then you cannot put the RDM on SCSI 0:1, or SCSI 0:2. You must put the RDM on SCSI 1:x or SCSI 2:x.

Matching LUN ID

Another reason for the above error is when the RDM is presented to the different ESXi hosts using a different LUN ID. The RDM must be presented to all ESXi hosts (and thus all MSCS nodes) using the same LUN ID.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage

IOOE

First off, there are a number of supported arrays from GreenBytes on the VMware HCL, namely their older GB series of arrays (which are no longer actively marketed) and their HA2200 model. However, their newer IOOE – IO Offload Engine – has recently appeared on our HCL, too. We talked a little bit about the IOOE, and where GreenBytes positions this product. IOOE is aimed squarely at the VDI market, but with a scale out aspect. It can scale starting at 1,000 VDI desktops, but can move up to 4,500 seats for a fully populated array. The scalability is based on the SSD capacity of the array. If additional seats over and above 4,500 are required, a new array can be deployed.

Protocols

IOOE presents LUNs to ESXi host over 10Gb iSCSI or 8Gb Fiber Channel (however, checking the HCL, it looks like it may be only qualified for iSCSI at the moment, with FC coming down the line).

VAAI Support

GreenBytes IOOE offers full support for the complete set of block primitives (Block Zero,Full Copy, Hardware Assisted Locking (ATS) & Thin Provisioning). This is always good to see.

Architecture

High Availability in the IOOE is provided by redundant and hot swappable components such as drives, fans and power supplies, as well as completely redundant storage controllers. Every component is replicated with dual heads & dual paths. GreenBytes also use patented RAID technology (which is a mixture of RAID 0 and RAID 6). So basically full redundancy of all components in the array.

 Configuration

As mentioned, everything is redundant, so the IOOE ships in a 6U format. This includes 2 x 2U controller chassis and a 1 x 2U SAS drive enclosure chassis which can hold up to 24 x 400GB SSD.

Inline Deduplication and Compression

This is the one aspect that the guys wanted to call out as being particularly unique to their IOOE storage solution. Using their patented zero latency inline deduplication technology, GreenBytes can reduce the space consumed by linked clones by up to 80%, and up to 97% for full clones. OK, so we’ve heard excellent compression/dedupe figures from other array vendors before, what makes this so special? Well, in this case, the dedupe is done in a single machine instruction, implying 0  latency incurred. This is why GreenBytes tell their VDI customers that they can use full clones instead of linked clones – they get great space savings with no impact to the I/O.

Snapshots

GreenBytes support snapshots at the volume level which are fully deduplicated for space-saving.

Replication & DR
Replication is also done at the volume level. There are a few neat things which GreenBytes does from a replication perspective, such as the ability to do many to 1 replication to a single target. They also have a patented deduplicated replication, where they only replicate unique changed blocks.

Unfortunately, there is no integration with Site Recovery Manager at this time for DR, but according to Steve (CEO), this certification is imminent with a GreenBytes SRA in the works. GreenBytes is working with VMware to get this completed ASAP.

Management

There is no vSphere Management plugin to manage GreenBytes from the vSphere UI. This is a shame, and I hope GreenBytes will consider this going forward as the ability to manage both the virtual infrastructure and storage infrastructure from a single management console has huge benefits for administrators.

Differentiators

I asked Steve what it was that made IOOE unique in a market where lots of storage vendors are positioning themselves as the solution to the VDI storage problem. The first thing which Steve mentioned was that because of their patented dedupe technology, customers can feel confident is using fully cloned desktops rather than using VMware linked clones when provisioning desktops. Steve added that linked clones may still be used, but I think customers would personally like to use full clones over linked clones if they are consuming what amounts to the same amount of physical disk space on the array.

So what about vIO?

Well, that’s the hardware product taken care of. What about this vIO product that was mentioned at the beginning of the article. Well, for all intents and purpose, vIO can be considered as providing all of the functionality that the IOOE has, but is deployed in the form of a storage appliance. In a nutshell, vIO can be considered as a virtual storage appliance which uses some flash storage to divert boot storms and swapping away from SAN, improving the overall performance of storage in a VDI environment.

The vIO appliance is designed to run on flash (which could be PCIe, local SSD or flash on an external array). The vIO virtual storage appliance takes storage mapped from the back-end storage array, and presents datastores (NFS, if I am not mistaken) to the hypervisor. Any virtual desktops deployed to the NFS datastore can now leverage the I/O acceleration and dedupe/compression capabilities of the vIO virtual storage appliance. Currently, vIO only works on the VMware ESXi hypervisor at this time.

How is vIO sold?
Steve informed me that vIO is licensing in 100 seats increments, so clearly aimed at the small to mid-size VDI markets. Customers requiring larger seat counts should consider the physical IOOE product.

Conclusion

GreenBytes are doing some very cool things at the moment. It is very interesting to see more movement in I/O acceleration via virtual appliance space – using virtual storage appliances with some ‘secret sauce’ to accelerate the I/O in VDI deployments seems to be resonating with folks as a low-cost way of handling storage performance issues.

While GreenBytes have a lot of decent integration points, it will be even better when there is a complete DR solution using SRM. It would also be nice to see management integration with the vSphere client.

But for those of you contemplating a VDI roll-out, GreenBytes may have products to help you on the journey. What’s nice is that you can start out small, and scale as necessary, either with IOOE or with vIO.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage