Photon Controller v1.1 and vSAN interoperability

PHOTON_square140Many of you will have seen the recent announcement for Photon Controller version 1.1. For me, the interesting part of this announcement is the support for vSAN as a storage platform with Photon Controller v1.1. I should think that the first question that those of you are familiar with both vSAN and Photon Controller will ask is “how do I configure vSAN for Photon Controller when there is no vCenter server in the mix?”. This is a very good question, and one which I will highlight in this blog post. There are also a few line items in the release notes which may not be very clear to those of you who are not very familiar with these products. I hope to be able to highlight these to you as well.

Deploying vSAN in Photon Controller

Now since there is no vCenter, there are two appliances that need to be deployed in a Photon Controller environment to enable you to successfully deploy vSAN. The first of these is a vSAN Manager appliance and the second is a Lightwave server for authentication. We then use RVC, the Ruby vSphere Console, which is shipped in the vSAN manager appliance, to setup vSAN.

Deploy a Lightwave server appliance

Let’s start with the Lightwave server. This is required, as we need to be able to authenticate who can set up (and more importantly, remove)  vSAN. I deployed VMware’s own Photon OS appliance for this task. Once deployed, I verified that DNS and hostname lookup was fully functional (a requirement) and I then installed the Lightwave server components:

root@LW-photon [ ~ ]# tdnf install -y vmware-lightwave-server

Once installed, I configured this Lightwave server with a domain and password, as follows.

root@LW-photon [ ~ ]# /opt/vmware/bin/configure-lightwave-server \
--domain rainpole.com --password *****

Deploying in standalone mode...
-----Begin installing components-----
VMware Authentication Framework
VMware Identity Manager
VMware Secure Token Service
Lightwave UI

Begin installing component: vmware-authentication-framework
139738022557336:error:0906D06C:lib(9):func(109):reason(108):pem_lib.c:701:\
Expecting: CERTIFICATE
Installed vmware-authentication-framework succesfully.

Begin installing component: vmware-identity-manager
Installed vmware-identity-manager succesfully.

Begin installing component: vmware-secure-token-service
Installed vmware-secure-token-service succesfully.

Begin installing component: lightwave-ui
Installed lightwave-ui succesfully.
Installation completed successfully.
Setup complete.
root@LW-photon [ ~ ]#

Once complete, the list of nodes should look similar to the following:

root@LW-photon [ ~ ]# cd /opt/vmware/bin
root@LW-photon [ /opt/vmware/bin ]# ./dir-cli nodes list
Enter password for administrator@rainpole.com:
Node: LW-photon.rainpole.com
Type: PSC
Site: Default-first-site

root@LW-photon [ /opt/vmware/bin ]#

The next step is to add a group and user which can be used for authentication when logging in to the vSAN manager appliance. These are the steps.

root@lw-photon [ /opt/vmware/bin ]# ./dir-cli ssogroup create --name VSANAdmins
Enter password for administrator@rainpolelw.com:
Service [VSANAdmins] created successfully

root@lw-photon [ /opt/vmware/bin ]# ./dir-cli user create --account vsanadmin \
--user-password '*****' --first-name vsanadmin --last-name vsanadmin
Enter password for administrator@rainpolelw.com:
User account [vsanadmin] created successfully

root@lw-photon [ /opt/vmware/bin ]# ./dir-cli group modify --name VSANAdmins \
--add vsanadmin
Enter password for administrator@rainpolelw.com:
Account [vsanadmin] added to group [VSANAdmins]
Group member [vsanadmin] added successfully

root@lw-photon [ /opt/vmware/bin ]# ./dir-cli group list --name VSANAdmins
Enter password for administrator@rainpolelw.com:
CN=vsanadmin vsanadmin,cn=users,dc=rainpolelw,dc=com
root@lw-photon [ /opt/vmware/bin ]#

By the way, the Lightwave server also comes with a UI, so the above steps of creating group, user and then adding the user to the group could have been accomplished this way instead:

lw-uiDeploy a vSAN manager appliance

The vSAN Manager is next be deployed. It comes as a OVF, so depending where it is being deployed from, it can be deployed via UI or OVF tool. I deployed it to a management infrastructure outside of Photon Controller, so used the web client UI. These are the fields that need to be populated:

vsan-ovaNote the Lightwave information that is required. You will need the IP address of the server, the admin password, the domain, and of course the authentication group that you created earlier. Once the vSAN manager has been deployed, you should be able to go back onto the Lightwave server, and check the node list. The vSAN manager should now be on the list of nodes:

root@lw-photon [ /opt/vmware/bin ]# ./dir-cli nodes list
Enter password for administrator@rainpolelw.com:
Node: lw-photon.rainpolelw.com
Type: PSC
Site: Default-first-site

Node: vsan-mgmt-srvr.rainpole.com
Type: Management

root@lw-photon [ /opt/vmware/bin ]#

Now comes the moment of truth. Can we successfully launch RVC, the Ruby vSphere Console, from the vSAN manager appliance and create a vSAN cluster?

Setup vSAN via RVC

Login in to the vSAN manager appliance, and verify that you can connect the RVC using the VSANAdmin credentials we created earlier in Lightwave. Note the syntax is rvc@domain@vsan-appliance:port.

root@vsan-mgmt-srvr [ ~ ]# rvc \
 vsanadmin@rainpole.com@vsan-mgmt-srvr.rainpole.com:8006
Install the "ffi" gem for better tab completion.
password:*****
0 /
1 vsan-mgmt-srvr.rainpole.com/
>

If successful, we can begin to create vSAN. If not successful, you need to revisit the previous steps, ensuring that the user and group are created successfully, and that the vSAN manager has successfully joined the Lightwave domain.

Assuming that the ESXi hosts have a VMkernel interface with vSAN traffic enabled, you can now got ahead and create a vSAN cluster as follows using RVC:

0 /
1 vsan-mgmt-srvr.rainpole.com/
> cd 1
/vsan-mgmt-srvr.rainpole.com> ls
0 Global (datacenter)
/vsan-mgmt-srvr.rainpole.com> cd 0
/vsan-mgmt-srvr.rainpole.com/Global> ls
0 storage/
1 vms [vmFolder-datacenter-1]/
2 datastores [datastoreFolder-datacenter-1]/
3 networks [networkFolder-datacenter-1]/
4 computers [hostFolder-datacenter-1]/

/vsan-mgmt-srvr.rainpole.com/Global> cd 4
/vsan-mgmt-srvr.rainpole.com/Global/computers> ls
/vsan-mgmt-srvr.rainpole.com/Global/computers> cluster.create demo
/vsan-mgmt-srvr.rainpole.com/Global/computers> ls
0 demo (cluster): cpu 0 GHz, memory 0 GB
/vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.cluster_change_autoclaim 0 -e
: success
No host specified to query, stop current operation.
/vsan-mgmt-srvr.rainpole.com/Global/computers> cluster.add_host 0 \
10.27.51.5 10.27.51.6 10.27.51.7 -u root -p *****
: success
: success
: success
/vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.enable_vsan_on_cluster 0

At this point, the cluster has been created, and hopefully the physical storage devices have been auto-claimed, and now you can go ahead and enable vSAN on the cluster, as per the instruction above. If all of the above was successful, you can now begin to use other RVC commands to look at objects, state of the cluster and the health of the cluster. For example, we can look at the cluster info:

/vsan-mgmt-srvr.rainpole.com/Global/computers> ls
0 demo (cluster): cpu 0 GHz, memory 0 GB
/vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.cluster_info 0
2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-05.rainpole.com (may take a moment) ...
2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-06.rainpole.com (may take a moment) ...
2016-11-23 09:03:44 +0000: Fetching host info from esxi-hp-07.rainpole.com (may take a moment) ...
Host: esxi-hp-05.rainpole.com
  Product: VMware ESXi 6.0.0 build-3620759
  VSAN enabled: yes
  Cluster info:
    Cluster role: master
    Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34
    Node UUID: 569c86cd-19dd-0723-8962-a0369f30c548
    Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \
 "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3)
  Node evacuated: no
  Storage info:
    Auto claim: yes
    Disk Mappings:
      SSD: HP Serial Attached SCSI Disk (naa.600508b1001cbbbe903bd48c8f6b2ddb) - 186 GB, v3
      MD: HP Serial Attached SCSI Disk (naa.600508b1001cade4bae23c7a79749263) - 745 GB, v3
  FaultDomainInfo:
    Not configured
  NetworkInfo:
    Adapter: vmk2 (192.50.0.5)

Host: esxi-hp-06.rainpole.com
  Product: VMware ESXi 6.0.0 build-3620759
  VSAN enabled: yes
  Cluster info:
    Cluster role: backup
    Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34
    Node UUID: 569ca570-06ce-1870-391b-a0369f56ddbc
    Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \
 "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3)
  Node evacuated: no
  Storage info:
    Auto claim: yes
    Disk Mappings:
      SSD: HP Serial Attached SCSI Disk (naa.600508b1001c81c97cf5080e8206c431) - 186 GB, v3
      MD: HP Serial Attached SCSI Disk (naa.600508b1001c51535caab0e295c86a91) - 745 GB, v3
  FaultDomainInfo:
    Not configured
  NetworkInfo:
    Adapter: vmk2 (192.50.0.6)

Host: esxi-hp-07.rainpole.com
  Product: VMware ESXi 6.0.0 build-3620759
  VSAN enabled: yes
  Cluster info:
    Cluster role: agent
    Cluster UUID: 3955b8aa-accb-443b-bfdb-8dc357a93e34
    Node UUID: 569c96d6-1dd9-d204-ac17-a0369f56dd10
    Member UUIDs: ["569c86cd-19dd-0723-8962-a0369f30c548", \
 "569ca570-06ce-1870-391b-a0369f56ddbc", "569c96d6-1dd9-d204-ac17-a0369f56dd10"] (3)
  Node evacuated: no
  Storage info:
    Auto claim: yes
    Disk Mappings:
      SSD: HP Serial Attached SCSI Disk (naa.600508b1001cc5956fa4ceab9c0f3840) - 186 GB, v3
      MD: HP Serial Attached SCSI Disk (naa.600508b1001c357b9abfce4730e1b697) - 745 GB, v3
  FaultDomainInfo:
    Not configured
  NetworkInfo:
    Adapter: vmk2 (192.50.0.7)

No Fault Domains configured in this cluster
/vsan-mgmt-srvr.rainpole.com/Global/computers>

And we can also look at the cluster health.

/vsan-mgmt-srvr.rainpole.com/Global/computers> ls
0 demo (cluster): cpu 0 GHz, memory 0 GB
/vsan-mgmt-srvr.rainpole.com/Global/computers> vsan.health.health_summary 0
Overall health: red (Virtual SAN HCL warning)
+-------------------------------------------------------------------------+---------+
| Health check | Result |
+-------------------------------------------------------------------------+---------+
| Hardware compatibility | Error |
| Virtual SAN HCL DB up-to-date | Error |
| Virtual SAN HCL DB Auto Update | Passed |
| SCSI Controller on Virtual SAN HCL | Warning |
| Controller Release Support | Warning |
| Controller Driver | Warning |
| Controller Firmware | Passed |
+-------------------------------------------------------------------------+---------+
| Performance service | Warning |
| Stats DB object | Warning |
+-------------------------------------------------------------------------+---------+
| Network | Passed |
| Hosts disconnected from VC | Passed |
| Hosts with connectivity issues | Passed |
| Virtual SAN cluster partition | Passed |
| Unexpected Virtual SAN cluster members | Passed |
| Hosts with Virtual SAN disabled | Passed |
| All hosts have a Virtual SAN vmknic configured | Passed |
| All hosts have matching subnets | Passed |
| All hosts have matching multicast settings | Passed |
| Multicast assessment based on other checks | Passed |
| Virtual SAN: Basic (unicast) connectivity check | Passed |
| Virtual SAN: MTU check (ping with large packet size) | Passed |
| vMotion: Basic (unicast) connectivity check | Passed |
| vMotion: MTU check (ping with large packet size) | Passed |
+-------------------------------------------------------------------------+---------+
| Physical disk | Passed |
| Overall disks health | Passed |
| Metadata health | Passed |
| Disk capacity | Passed |
| Software state health | Passed |
| Congestion | Passed |
| Component limit health | Passed |
| Component metadata health | Passed |
| Memory pools (heaps) | Passed |
| Memory pools (slabs) | Passed |
+-------------------------------------------------------------------------+---------+
| Cluster | Passed |
| ESX Virtual SAN Health service installation | Passed |
| Virtual SAN Health Service up-to-date | Passed |
| Advanced Virtual SAN configuration in sync | Passed |
| Virtual SAN CLOMD liveness | Passed |
| Virtual SAN Disk Balance | Passed |
| Deduplication and compression configuration consistency | Passed |
| Disk group with incorrect deduplication and compression configuration | Passed |
| Software version compatibility | Passed |
| Disk format version | Passed |
+-------------------------------------------------------------------------+---------+
| Limits | Passed |
| Current cluster situation | Passed |
| After 1 additional host failure | Passed |
| Host component limit | Passed |
+-------------------------------------------------------------------------+---------+

Details about any failed test below ...
Hardware compatibility - Virtual SAN HCL DB up-to-date: red
 +--------------------------------+---------------------+
 | Entity | Time in UTC |
 +--------------------------------+---------------------+
 | Current time | 2016-11-22 07:36:49 |
 | Local HCL DB copy last updated | 2016-01-12 14:10:54 |
 +--------------------------------+---------------------+

Hardware compatibility - SCSI Controller on Virtual SAN HCL: yellow
 +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+
 | Host | Device | Display Name | Used by Virtual SAN | PCI ID | On HCL |
 +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+
 | esxi-hp-05.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning |
 | esxi-hp-07.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning |
 | esxi-hp-06.rainpole.com | vmhba1 | Hewlett-Packard Company Smart Array P410i | Yes | 103c ,323a,103c,3245 | Warning |
 +-------------------------+--------+-------------------------------------------+---------------------+----- ----------------+---------+

Hardware compatibility - Controller Release Support: yellow
 +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+
 | Host | Device | Release of ESX | Release supported | Releases on HCL |
 +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+
 | esxi-hp-05.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A |
 | esxi-hp-07.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A |
 | esxi-hp-06.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | ESXi 6.0 U2 | Warning | N/A |
 +-------------------------+---------------------------------------------------+----------------+----------- --------+-----------------+

Hardware compatibility - Controller Driver: yellow
 +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+
 | Host | Device | Driver in use | Driver health | Drivers on HCL |
 +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+
 | esxi-hp-05.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A |
 | esxi-hp-07.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A |
 | esxi-hp-06.rainpole.com | vmhba1: Hewlett-Packard Company Smart Array P410i | hpsa (6.0.0.44-4vmw) | Warn ing | N/A |
 +-------------------------+---------------------------------------------------+----------------------+----- ----------+----------------+

Performance service - Stats DB object: yellow
 +---------------+---------+--------+
 | Check | Result | Status |
 +---------------+---------+--------+
 | Object exists | Warning | No |
 +---------------+---------+--------+

[[4.191779542, "initial connect"],
 [41.828061056, "cluster-health"],
 [0.009371718, "table-render"]]
/vsan-mgmt-srvr.rainpole.com/Global/computers>

OK, so this health summary has highlighted a lot of issues with my vSAN cluster, especially with regards to driver and controller (you can scroll right to see more). I should address this before doing anything with this environment from a production perspective, but hopefully it gives you the idea.

Release notes review

The one thing which is an issue at the moment, and which is called out in the release notes, is the fact that the performance service is not working properly in this release.

To add clarity to some of the other statements in the release notes, notably the detection of the vSAN datastore 15 minutes after it has been created, or consuming it via the photon controller yml file, we did not want to put any restrictions on the user about when to install vSAN. So basically you can set this up in two ways. The first is to install Photon Controller first, then vSAN or install vSAN first, then Photon Controller. If a vSAN datastore is created after Photon Controller is installed, Photon Controller detects that vSAN datastore after 15 minutes. This means that if you installed Photon Controller first, then created a cluster, Photon Controller will detect it but 15 minutes later. This datastore can then be used for the provisioning of VMs (for clustering frameworks like Kubernetes).  If you are planning to use a vSAN datastore as the image datastore however, but you plan to install vSAN after Photon Controller, you must make sure that you add vsanDatastore in the deployment yml. There is a known issue, which will be addressed in a future release, where we can’t easily use the vsanDatastore if we don’t know about it at deployment time.

I’m going to leave this here for now, but I will follow-up shortly with a post on how we can deploy Kubernetes onto this configuration using photon controller CLI, placing the Kubernetes VMs (master, etcd, workers) on the vSAN datastore.

5 comments
  1. Great article, I have really enjoyed your article. You show how to deploy Photon Controller v1.1 with vSAN . It is really helpful. I have done by the help of your article. Thanks for sharing. The way you explained each and everything is really great. Thanks once again for sharing.

  2. It’s a nice post which is just what I’m looking for.

    I deployed a VSAN Manager Appliance on a standalone ESXi using ovftool, but the host name and root password of the VM are always disregarded, and the VSAN Manager appliance is not listed in dir-cli nodes list. Any advices?

    Thanks.

    • I’m afraid not. Check the issues section on Github to see if anyone has encountered it before. Otherwise raise it as a new issue if you can’t find a solution.

Comments are closed.