Deploying vSAN in Photon Controller
Now since there is no vCenter, there are two appliances that need to be deployed in a Photon Controller environment to enable you to successfully deploy vSAN. The first of these is a vSAN Manager appliance and the second is a Lightwave server for authentication. We then use RVC, the Ruby vSphere Console, which is shipped in the vSAN manager appliance, to setup vSAN.
Deploy a Lightwave server appliance
Let’s start with the Lightwave server. This is required, as we need to be able to authenticate who can set up (and more importantly, remove) vSAN. I deployed VMware’s own Photon OS appliance for this task. Once deployed, I verified that DNS and hostname lookup was fully functional (a requirement) and I then installed the Lightwave server components:
root@LW-photon [ ~ ]# tdnf install -y vmware-lightwave-server
Once installed, I configured this Lightwave server with a domain and password, as follows.
root@LW-photon [ ~ ]# /opt/vmware/bin/configure-lightwave-server \ --domain rainpole.com --password ***** Deploying in standalone mode... -----Begin installing components----- VMware Authentication Framework VMware Identity Manager VMware Secure Token Service Lightwave UI Begin installing component: vmware-authentication-framework 139738022557336:error:0906D06C:lib(9):func(109):reason(108):pem_lib.c:701:\ Expecting: CERTIFICATE Installed vmware-authentication-framework succesfully. Begin installing component: vmware-identity-manager Installed vmware-identity-manager succesfully. Begin installing component: vmware-secure-token-service Installed vmware-secure-token-service succesfully. Begin installing component: lightwave-ui Installed lightwave-ui succesfully. Installation completed successfully. Setup complete. root@LW-photon [ ~ ]#
Once complete, the list of nodes should look similar to the following:
root@LW-photon [ ~ ]# cd /opt/vmware/bin root@LW-photon [ /opt/vmware/bin ]# ./dir-cli nodes list Enter password for administrator@rainpole.com: Node: LW-photon.rainpole.com Type: PSC Site: Default-first-site root@LW-photon [ /opt/vmware/bin ]#
The next step is to add a group and user which can be used for authentication when logging in to the vSAN manager appliance. These are the steps.
root@lw-photon [ /opt/vmware/bin ]# ./dir-cli ssogroup create --name VSANAdmins Enter password for administrator@rainpolelw.com: Service [VSANAdmins] created successfully root@lw-photon [ /opt/vmware/bin ]# ./dir-cli user create --account vsanadmin \ --user-password '*****' --first-name vsanadmin --last-name vsanadmin Enter password for administrator@rainpolelw.com: User account [vsanadmin] created successfully root@lw-photon [ /opt/vmware/bin ]# ./dir-cli group modify --name VSANAdmins \ --add vsanadmin Enter password for administrator@rainpolelw.com: Account [vsanadmin] added to group [VSANAdmins] Group member [vsanadmin] added successfully root@lw-photon [ /opt/vmware/bin ]# ./dir-cli group list --name VSANAdmins Enter password for administrator@rainpolelw.com: CN=vsanadmin vsanadmin,cn=users,dc=rainpolelw,dc=com root@lw-photon [ /opt/vmware/bin ]#
By the way, the Lightwave server also comes with a UI, so the above steps of creating group, user and then adding the user to the group could have been accomplished this way instead:
The vSAN Manager is next be deployed. It comes as a OVF, so depending where it is being deployed from, it can be deployed via UI or OVF tool. I deployed it to a management infrastructure outside of Photon Controller, so used the web client UI. These are the fields that need to be populated:
root@lw-photon [ /opt/vmware/bin ]# ./dir-cli nodes list Enter password for administrator@rainpolelw.com: Node: lw-photon.rainpolelw.com Type: PSC Site: Default-first-site Node: vsan-mgmt-srvr.rainpole.com Type: Management root@lw-photon [ /opt/vmware/bin ]#
Now comes the moment of truth. Can we successfully launch RVC, the Ruby vSphere Console, from the vSAN manager appliance and create a vSAN cluster?
Setup vSAN via RVC
Login in to the vSAN manager appliance, and verify that you can connect the RVC using the VSANAdmin credentials we created earlier in Lightwave. Note the syntax is rvc@domain@vsan-appliance:port.
root@vsan-mgmt-srvr [ ~ ]# rvc \ vsanadmin@rainpole.com@vsan-mgmt-srvr.rainpole.com:8006 Install the "ffi" gem for better tab completion. password:***** 0 / 1 vsan-mgmt-srvr.rainpole.com/ >
If successful, we can begin to create vSAN. If not successful, you need to revisit the previous steps, ensuring that the user and group are created successfully, and that the vSAN manager has successfully joined the Lightwave domain.
Assuming that the ESXi hosts have a VMkernel interface with vSAN traffic enabled, you can now got ahead and create a vSAN cluster as follows using RVC:
0 / 1 vsan-mgmt-srvr.rainpole.com/ > cd 1 /vsan-mgmt-srvr.rainpole.com> ls 0 Global (datacenter) /vsan-mgmt-srvr.rainpole.com> cd 0 /vsan-mgmt-srvr.rainpole.com/Global> ls 0 storage/ 1 vms [vmFolder-datacenter-1]/ 2 datastores [datastoreFolder-datacenter-1]/ 3 networks [networkFolder-datacenter-1]/ 4 computers [hostFolder-datacenter-1]/ /vsan-mgmt-srvr.rainpole.com/Global> cd 4 /vsan-mgmt-srvr.rainpole.com/Global/computers> ls /vsan-mgmt-srvr.rainpole.com/Global/computers> cluster.create demo /vsan-mgmt-srvr.rainpole.com/Global/computers> ls 0 demo (cluster): cpu 0 GHz, memory 0 GB /vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.cluster_change_autoclaim 0 -e : success No host specified to query, stop current operation. /vsan-mgmt-srvr.rainpole.com/Global/computers> cluster.add_host 0 \ 10.27.51.5 10.27.51.6 10.27.51.7 -u root -p ***** : success : success : success /vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.enable_vsan_on_cluster 0
At this point, the cluster has been created, and hopefully the physical storage devices have been auto-claimed, and now you can go ahead and enable vSAN on the cluster, as per the instruction above. If all of the above was successful, you can now begin to use other RVC commands to look at objects, state of the cluster and the health of the cluster. For example, we can look at the cluster info:
/vsan-mgmt-srvr.rainpole.com/Global/computers> ls 0 demo (cluster): cpu 0 GHz, memory 0 GB /vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.cluster_info 0 2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-05.rainpole.com (may take a moment) ... 2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-06.rainpole.com (may take a moment) ... 2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-07.rainpole.com (may take a moment) ... Host: esxi-hp-05.rainpole.com Product: VMware ESXi 6.0.0 build-3620759 VSAN enabled: yes Cluster info: Cluster role: master Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34 Node UUID: 569c86cd-19dd-0723-8962-a0369f30c548 Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \ "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3) Node evacuated: no Storage info: Auto claim: yes Disk Mappings: SSD: HP Serial Attached SCSI Disk (naa.600508b1001cbbbe903bd48c8f6b2ddb) - 186 GB, v3 MD: HP Serial Attached SCSI Disk (naa.600508b1001cade4bae23c7a79749263) - 745 GB, v3 FaultDomainInfo: Not configured NetworkInfo: Adapter: vmk2 (192.50.0.5) Host: esxi-hp-06.rainpole.com Product: VMware ESXi 6.0.0 build-3620759 VSAN enabled: yes Cluster info: Cluster role: backup Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34 Node UUID: 569ca570-06ce-1870-391b-a0369f56ddbc Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \ "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3) Node evacuated: no Storage info: Auto claim: yes Disk Mappings: SSD: HP Serial Attached SCSI Disk (naa.600508b1001c81c97cf5080e8206c431) - 186 GB, v3 MD: HP Serial Attached SCSI Disk (naa.600508b1001c51535caab0e295c86a91) - 745 GB, v3 FaultDomainInfo: Not configured NetworkInfo: Adapter: vmk2 (192.50.0.6) Host: esxi-hp-07.rainpole.com Product: VMware ESXi 6.0.0 build-3620759 VSAN enabled: yes Cluster info: Cluster role: agent Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34 Node UUID: 569c96d6-1dd9-d204-ac17-a0369f56dd10 Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \ "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3) Node evacuated: no Storage info: Auto claim: yes Disk Mappings: SSD: HP Serial Attached SCSI Disk (naa.600508b1001cc5956fa4ceab9c0f3840) - 186 GB, v3 MD: HP Serial Attached SCSI Disk (naa.600508b1001c357b9abfce4730e1b697) - 745 GB, v3 FaultDomainInfo: Not configured NetworkInfo: Adapter: vmk2 (192.50.0.7) No Fault Domains configured in this cluster /vsan-mgmt-srvr.rainpole.com/Global/computers>
And we can also look at the cluster health.
/vsan-mgmt-srvr.rainpole.com/Global/computers> ls 0 demo (cluster): cpu 0 GHz, memory 0 GB /vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.health.health_summary 0 Overall health: red (Virtual SAN HCL warning) +-------------------------------------------------------------------------+---------+ | Health check | Result | +-------------------------------------------------------------------------+---------+ | Hardware compatibility | Error | | Virtual SAN HCL DB up-to-date | Error | | Virtual SAN HCL DB Auto Update | Passed | | SCSI Controller on Virtual SAN HCL | Warning | | Controller Release Support | Warning | | Controller Driver | Warning | | Controller Firmware | Passed | +-------------------------------------------------------------------------+---------+ | Performance service | Warning | | Stats DB object | Warning | +-------------------------------------------------------------------------+---------+ | Network | Passed | | Hosts disconnected from VC | Passed | | Hosts with connectivity issues | Passed | | Virtual SAN cluster partition | Passed | | Unexpected Virtual SAN cluster members | Passed | | Hosts with Virtual SAN disabled | Passed | | All hosts have a Virtual SAN vmknic configured | Passed | | All hosts have matching subnets | Passed | | All hosts have matching multicast settings | Passed | | Multicast assessment based on other checks | Passed | | Virtual SAN: Basic (unicast) connectivity check | Passed | | Virtual SAN: MTU check (ping with large packet size) | Passed | | vMotion: Basic (unicast) connectivity check | Passed | | vMotion: MTU check (ping with large packet size) | Passed | +-------------------------------------------------------------------------+---------+ | Physical disk | Passed | | Overall disks health | Passed | | Metadata health | Passed | | Disk capacity | Passed | | Software state health | Passed | | Congestion | Passed | | Component limit health | Passed | | Component metadata health | Passed | | Memory pools (heaps) | Passed | | Memory pools (slabs) | Passed | +-------------------------------------------------------------------------+---------+ | Cluster | Passed | | ESX Virtual SAN Health service installation | Passed | | Virtual SAN Health Service up-to-date | Passed | | Advanced Virtual SAN configuration in sync | Passed | | Virtual SAN CLOMD liveness | Passed | | Virtual SAN Disk Balance | Passed | | Deduplication and compression configuration consistency | Passed | | Disk group with incorrect deduplication and compression configuration | Passed | | Software version compatibility | Passed | | Disk format version | Passed | +-------------------------------------------------------------------------+---------+ | Limits | Passed | | Current cluster situation | Passed | | After 1 additional host failure | Passed | | Host component limit | Passed | +-------------------------------------------------------------------------+---------+ Details about any failed test below ... Hardware compatibility - Virtual SAN HCL DB up-to-date: red +--------------------------------+---------------------+ | Entity | Time in UTC | +--------------------------------+---------------------+ | Current time | 2016-11-22 07:36:49 | | Local HCL DB copy last updated | 2016-01-12 14:10:54 | +--------------------------------+---------------------+ Hardware compatibility - SCSI Controller on Virtual SAN HCL: yellow +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+ | Host | Device | Display Name | Used by Virtual SAN | PCI ID | On HCL | +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+ | esxi-hp-05.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning | | esxi-hp-07.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning | | esxi-hp-06.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning | +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+ Hardware compatibility - Controller Release Support: yellow +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+ | Host | Device | Release of ESX | Release supported | Releases on HCL | +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+ | esxi-hp-05.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A | | esxi-hp-07.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A | | esxi-hp-06.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A | +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+ Hardware compatibility - Controller Driver: yellow +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+ | Host | Device | Driver in use | Driver health | Drivers on HCL | +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+ | esxi-hp-05.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A | | esxi-hp-07.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A | | esxi-hp-06.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A | +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+ Performance service - Stats DB object: yellow +---------------+---------+--------+ | Check | Result | Status | +---------------+---------+--------+ | Object exists | Warning | No | +---------------+---------+--------+ [[4.191779542, "initial connect"], [41.828061056, "cluster-health"], [0.009371718, "table-render"]] /vsan-mgmt-srvr.rainpole.com/Global/computers>
OK, so this health summary has highlighted a lot of issues with my vSAN cluster, especially with regards to driver and controller (you can scroll right to see more). I should address this before doing anything with this environment from a production perspective, but hopefully it gives you the idea.
Release notes review
The one thing which is an issue at the moment, and which is called out in the release notes, is the fact that the performance service is not working properly in this release.
To add clarity to some of the other statements in the release notes, notably the detection of the vSAN datastore 15 minutes after it has been created, or consuming it via the photon controller yml file, we did not want to put any restrictions on the user about when to install vSAN. So basically you can set this up in two ways. The first is to install Photon Controller first, then vSAN or install vSAN first, then Photon Controller. If a vSAN datastore is created after Photon Controller is installed, Photon Controller detects that vSAN datastore after 15 minutes. This means that if you installed Photon Controller first, then created a cluster, Photon Controller will detect it but 15 minutes later. This datastore can then be used for the provisioning of VMs (for clustering frameworks like Kubernetes). If you are planning to use a vSAN datastore as the image datastore however, but you plan to install vSAN after Photon Controller, you must make sure that you add vsanDatastore in the deployment yml. There is a known issue, which will be addressed in a future release, where we can’t easily use the vsanDatastore if we don’t know about it at deployment time.
I’m going to leave this here for now, but I will follow-up shortly with a post on how we can deploy Kubernetes onto this configuration using photon controller CLI, placing the Kubernetes VMs (master, etcd, workers) on the vSAN datastore.