VSAN Part 11 – Shutting down the VSAN cluster
In a post on the vSphere blog, I spoke about how to use maintenance mode. As a follow on request, a number of people asked me how they should safely shutdown a VSAN cluster. In this post, I will address that question and share my observations.
On my three-node VSAN cluster, I had a number of virtual machines as well as a vApp running vCenter Operations Manager VMs. My first step was to shut down all virtual machines in my cluster.
Once the virtual machines were powered down, I then proceeded to place my ESXi hosts into maintenance mode. When the maintenance mode wizard appeared, I made the following choices:
I unchecked the flag which specifies ‘Move powered-off and suspended virtual machines to other hosts in the cluster‘. The reason for this was because I was going to put all hosts into maintenance mode to shut down the cluster. Therefore there was no reason to migrate the virtual machines between hosts. I also elected to do no data migration for the exact same reason; there is no point in moving the virtual machine storage objects to other hosts/disks as I am shutting down the whole cluster. I put all three nodes into maintenance mode successfully.
With all virtual machines powered down, and all three ESXi hosts in maintenance mode, I then proceeded to reboot all three ESXi nodes. All hosts rebooted successfully, and of course, the VSAN cluster status was reporting issues with the configuration during this time as one might expect (no eligible disks and network issues):
Once the hosts rebooted, they reformed the cluster. I took each of them out of maintenance mode, and I was able to successfully restart all my virtual machines, including the vCOps VMs in the vApp. Note that at no point did I need to disable Virtual SAN on the cluster.
To recap, if shutting down the whole of the VSAN cluster, use maintenance mode for the hosts, and do not move VMs or migrate any data.
Hi Cormac,
What’s happens if your running vCenter on the VSAN Datastore?
Michael
I am also looking for a strategy for a clean shutdown and startup procedure when vCA is running from the VSAN..
When you say vCA, I guess you mean vCenter Server.
I haven’t actually went through that workflow, but I’m guessing you would need to move vCenter to local storage on one of the hosts or move it to shared storage to complete the clean shutdown procedure.
Thank you Cormac! That confirms my thinking with regards to a converged design: it would be best to still keep some local datastores around. I should have said vCSA, apologies for the mislabeling.
Cormac – Great post.
We are working to do an automated shutdown script using powershell and the CLI. Is there any chance you know if the CLI currently supports the “Virtual SAN data migration” option?
Here is the particular line that sets our hosts in maintenance mode but we can’t find any way to set the vSAN flag to keep the data from migrating:
Set-VMHost -VMHost $_.Name -State “Maintenance” -Evacuate:$false -RunAsync
Thanks.
I personally do not know, but I will ask some folks for you.
It’s not there at the moment, but the power I team have filed a feature request to get this in. Fingers crossed that we can get it in for GA.
Apparently there is a method available through the API.
$vmhost= Get-VMHost 1.2.3.4
$spec = new-object VMware.Vim.HostMaintenanceSpec
$spec.VsanMode = new-object VMware.Vim.VsanHostDecommissionMode
$spec.VsanMode.ObjectAction = “ensureObjectAccessibility”
$vmhost.ExtensionData.EnterMaintenanceMode(0, $false, $spec)
Can you test this and let me know if it works?
We were able to get this working but had to modify the VMware.Vim.VsanHostDecommissionMode object action to noAction. When set to ensureObjectAccessibilty it appeared vSAN was still working to move storage around. We did not have time to monitor the exact vSAN activity with the ruby vSphere console.
The VMware documentation for this is not exactly clear:
http://pubs.vmware.com/vsphere-55/index.jsp#com.vmware.wssdk.apiref.doc/vim.vsan.host.DecommissionMode.ObjectAction.html
I believe that as long as we have all VMs power off and do not intend to decommission the host noAction should be acceptable and equivalent to the vSphere Web interface. Can you confirm if this is an accurate statement?
As far as the other users’ questions regarding vCenter, we were able to successfully script the shutdown of all VMs, including vCenter, as well as dropping the hosts into maintenance mode. Our vCenter VM was running on our 3-node cluster but was not running on the vSAN storage. We are still working to figure out the best way to initiate the host shutdown commands to have a full, automated, and graceful shutdown of a vSAN cluster. Something that we’ve found to be a much more challenging than it should be. IMO VMware really ought to integrate some of these fundamental tasks into vCenter or the hosts.
Thanks for the help.