Site icon CormacHogan.com

What happens when VMFS heap depletes completely?

I’ve blogged about the VMFS heap situation numerous times now already. However, a question that I frequently get asked is what actual happens when heap runs out? I thought I’d put together a short article explaining the symptoms one would see when there is no VMFS heap left on an ESXi host. Thanks once again to my good friend and colleague, Paudie O’Riordan, for sharing his support experiences with me on this matter – “together we win”, right Paud?

An actual heap depletion message in the vmkernel.log looks similar to this:

WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
WARNING: Heap: 2900: Heap_Align(vmfs3, 524288/524288 bytes, 8 align) failed. caller: 0x418028a95e74

But what are the user visible symptoms?

The VM failed to resume on the destination during early power on. Reason: 0 (Cannot allocate memory).

Cannot open the disk ‘/vmfs/volumes/5106a125-91aedda0-17fb-0025b551a04f/IP-FS-507/IP-FS-507_1.vmdk’ or one of the snapshot disks it depends on”. 

Other errors observed include “A general system error occurred: The virtual machine could not start”.

As you can see, heap depletion can have serious side-effects in your environment. Therefore, if you are deploying a large number of very large VMDKs on your ESXi hosts(s), consider upgrading to ESXi 5.0p5 (released in March 2013) or 5.1U1 (released in April 2013). Both the default and the maximum size of the heap has been increased to 640MB. This means that the full 64TB of a VMFS volume may be addressed without the risk of heap depletion.

And to conclude, we are doing a lot of work in this area in our next release, and I will share much more details on how this issue will be a thing of the past as soon as I can.

Exit mobile version