Some time back, I started work on vTopology, which allows me to plugin a Shell/PowerShell script into a mechanism called krew and run it from kubectl to get some information. However it is a little cumbersome to get all the pieces in place. So I began looking at alternate ways in which I could achieve the same thing without requiring any external dependencies. Kubernetes Customer Resource Definitions (CRDs) and Operators seem to be universally recognized as the de-facto way to extend Kubernetes. Thus, I started to look at how I might be able to create a CRD and operator to query for vSphere objects such as HostInfo, or DiskInfo, or VMInfo and use these to query underlying vSphere resources from kubectl.
As a proof-of-concept, I built a very simple CRD and Operator which returns the TotalCPU and FreeCPU of an ESXi host. I learnt so much from trying to do this exercise that I decided to write up the steps on GitHub and share them with you. If you are looking to learn more about Kubernetes CRD and Operators, and are interested in how to get it to interact with VMware’s govmomi APIs provided by VMware for vSphere, you might like to check it out. It is a long way off from providing all of the detail of the underlying infrastructure which I currently have in vTopology today, but maybe over time I’ll be able to add more features.
The complete code for the operator and CRD can be found here on my GitHub repository: https://github.com/cormachogan/hostinfo-operator, along with step by step instructions on how to deploy it on your own Kubernetes cluster. Hope you find it useful. This was updated [18th Jan 2021] to move the vSphere login out of the Reconciler code and into main.go to avoid calling vSphere on every reconcile. Here is a sample output which contains CPU usage information in the status:
$ kubectl get hi -o yaml apiVersion: v1 items: - apiVersion: topology.corinternal.com/v1 kind: HostInfo metadata: creationTimestamp: "2021-01-18T14:15:07Z" generation: 1 managedFields: - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:spec: .: {} f:hostname: {} manager: kubectl operation: Update time: "2021-01-18T14:15:07Z" - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:freeCPU: {} f:totalCPU: {} manager: manager operation: Update time: "2021-01-18T14:31:00Z" name: hostinfo-host-e namespace: default resourceVersion: "28883011" selfLink: /apis/topology.corinternal.com/v1/namespaces/default/hostinfoes/hostinfo-host-e uid: 720a91bb-8929-4120-8ba9-d652c884f9ed spec: hostname: esxi-dell-e.rainpole.com status: freeCPU: 41238 totalCPU: 43980 kind: List metadata: resourceVersion: ""
I also created an operator to retrieve virtual machine information. You also find it on GitHub, here: https://github.com/cormachogan/vminfo-operator. Again, you can see the sorts of VM information that we can pull via the operator in the status fields.
$ kubectl get vminfo -o yaml apiVersion: v1 items: - apiVersion: topology.corinternal.com/v1 kind: VMInfo metadata: creationTimestamp: "2021-01-18T12:20:45Z" generation: 1 managedFields: - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:spec: .: {} f:nodename: {} manager: kubectl operation: Update time: "2021-01-18T12:20:45Z" - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:guestId: {} f:hwVersion: {} f:ipAddress: {} f:pathToVM: {} f:powerState: {} f:resvdCPU: {} f:resvdMem: {} f:totalCPU: {} f:totalMem: {} manager: manager operation: Update time: "2021-01-18T12:20:46Z" name: tkg-worker-1 namespace: default resourceVersion: "28841720" selfLink: /apis/topology.corinternal.com/v1/namespaces/default/vminfoes/tkg-worker-1 uid: 2c60b273-a866-4344-baf5-0b3b924b65a5 spec: nodename: tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298 status: guestId: vmwarePhoton64Guest hwVersion: vmx-17 ipAddress: 10.27.62.45 pathToVM: '[vsanDatastore] 4d56b55f-11db-8822-6463-246e962f4914/tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298.vmx' powerState: poweredOn resvdCPU: 0 resvdMem: 0 totalCPU: 2 totalMem: 4096 kind: List metadata: resourceVersion: "" selfLink: ""
My final exercise was to create a tutorial on how to get FCD information. FCDs, short for First Class Disks, are used to back Kubernetes Persistent Volumes when these are deployed on vSphere Storage using the vSphere CSI driver. The operator is here: https://github.com/cormachogan/fcdinfo-operator. Here is some of the information we can get for the PV / FCD, such as the path to the file, and the provisioning type (thick/thin):
$ kubectl get fcd -o yaml apiVersion: v1 items: - apiVersion: topology.corinternal.com/v1 kind: FCDInfo metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"topology.corinternal.com/v1","kind":"FCDInfo","metadata":{"annotations":{},"name":"fcdinfo-sample","namespace":"default"},"spec":{"pvId":"pvc-e3f6dd59-cbc0-49a7-97c8-d92a26732c43"}} creationTimestamp: "2021-01-26T10:43:21Z" generation: 1 managedFields: - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:spec: .: {} f:pvId: {} manager: kubectl operation: Update time: "2021-01-26T10:43:21Z" - apiVersion: topology.corinternal.com/v1 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:filePath: {} f:provisioningType: {} f:sizeMB: {} manager: manager operation: Update time: "2021-01-26T10:43:22Z" name: fcdinfo-sample namespace: default resourceVersion: "32818807" selfLink: /apis/topology.corinternal.com/v1/namespaces/default/fcdinfoes/fcdinfo-sample uid: 5d51788d-fc1b-441f-be11-723d02c87b4b spec: pvId: pvc-e3f6dd59-cbc0-49a7-97c8-d92a26732c43 status: filePath: '[vsanDatastore] 038f6b5f-8122-d3af-eabe-246e962c240c/b39bcacc6ff143439f9cd6b7454999e4.vmdk' provisioningType: thin sizeMB: 1024 kind: List metadata: resourceVersion: "" selfLink: ""
I learnt loads from building these operators. I hope you find the tutorials useful, both from an operator and a govmomi persepctive.