Why do I get “Error from server (Forbidden)” in vSphere with Tanzu
I’ve seen a number of queries around the behaviour of vSphere with Tanzu when it comes to querying Kubernetes objects on the Supervisor Cluster. More often than not, it is a question which arises when a user get an error similar to the following:
Error from server (Forbidden): wcpnamespaces.appplatform.wcp.vmware.com is forbidden: \
User "sso:Administrator@vsphere.local" cannot list resource "wcpnamespaces" in API group \
"appplatform.wcp.vmware.com" in the namespace "cormac-ns"
The reason for these errors is because the Supervisor Cluster is not treated as a general purpose Kubernetes cluster. The predominant role of the Supervisor Cluster is to provide services, such as the TKG Service for provisioning Kubernetes clusters and the VM Service for provisioning virtual machines. However, when troubleshooting the Supervisor cluster, errors like the one shown above may make you think something is wrong and may set you off on the wrong direction. Therefore, I provide a list of all API resources from my vSphere 8.0 environment below. I highlight which ones are query-able from the kubectl command when running in the Supervisor Cluster Default Namespace context, and which are query-able when running in a user-created vSphere Namespace context. This will hopefully help allay any fears that something is not correct on the Supervisor. The first column is the name of API resource, retrieved from the command kubectl api-resources. This has been sorted in alphabetical order, and only the name of the resource is displayed. The second column shows whether or not a kubectl get can be run against this resource from a vSphere Namespace context or if it is forbidden. The last column is a kubectl get run against the Supervisor Default Namespace, i.e. when the context is set to the Supervisor Cluster API Server, and again shows whether or not it is forbidden. Note that these outputs are mostly the same, but there are a handful of differences.
| API Resources | Forbidden? (vSphere Namespace context) | Forbidden? (SV Default Namespace context) |
| activedirectoryidentityproviders | YES | YES |
| agentinstalls | NO | NO |
| antreaconfigs | NO | NO |
| apiservices | YES | YES |
| apps | YES | YES |
| availabilityzones | YES | YES |
| aviloadbalancerconfigs | YES | YES |
| blockaffinities | YES | YES |
| calicoconfigs | NO | NO |
| capabilities | YES | YES |
| ccsplugins | NO | NO |
| certificaterequests | NO | NO |
| certificates | NO | NO |
| certificatesigningrequests | YES | YES |
| challenges | NO | NO |
| cliplugins | NO | NO |
| clusterbootstraps | NO | NO |
| clusterbootstraptemplates | NO | NO |
| clusterclasses | NO | NO |
| clusterissuers | YES | YES |
| clusterresourcesetbindings | YES | YES |
| clusterresourcesets | YES | YES |
| clusterrolebindings | YES | YES |
| clusterroles | YES | YES |
| clusters | NO | NO |
| cnscsisvfeaturestates | YES | YES |
| cnsnodevmattachments | NO | NO |
| cnsvolumemetadatas | YES | YES |
| cnsvolumeoperationrequests | YES | YES |
| compatibilities | YES | YES |
| componentstatuses | YES | YES |
| configmaps | NO | NO |
| contentlibraryproviders | NO | NO |
| contentsourcebindings | NO | NO |
| contentsources | NO | NO |
| controllerrevisions | NO | NO |
| credentialissuers | YES | YES |
| cronjobs | NO | NO |
| csidrivers | YES | YES |
| csinodes | YES | YES |
| csistoragecapacities | YES | YES |
| customresourcedefinitions | YES | YES |
| daemonsets | NO | NO |
| deployments | NO | NO |
| endpoints | NO | NO |
| endpointslices | NO | NO |
| events | NO | NO |
| featuregates | NO | NO |
| features | NO | NO |
| federationdomains | YES | YES |
| flowschemas | YES | YES |
| gatewayclasses | YES | YES |
| gateways | NO | NO |
| haproxyloadbalancerconfigs | YES | YES |
| horizontalpodautoscalers | NO | NO |
| httproutes | YES | YES |
| imagedisks | NO | NO |
| images | NO | NO |
| ingressclasses | YES | YES |
| ingresses | NO | NO |
| internalpackagemetadatas | YES | YES |
| internalpackages | YES | YES |
| ipamblocks | YES | YES |
| ipamconfigs | YES | YES |
| ipamhandles | YES | YES |
| ippools | YES | YES |
| issuers | NO | NO |
| jobs | NO | NO |
| jwtauthenticators | YES | YES |
| kappcontrollerconfigs | NO | NO |
| kubeadmconfigs | NO | NO |
| kubeadmconfigtemplates | NO | NO |
| kubeadmcontrolplanes | NO | NO |
| kubeadmcontrolplanetemplates | YES | YES |
| kuberneteslicenses | NO | NO |
| ldapidentityproviders | YES | YES |
| leases | YES | YES |
| limitranges | NO | NO |
| loadbalancerconfigs | YES | YES |
| machinedeployments | NO | NO |
| machinehealthchecks | NO | NO |
| machinepools | YES | YES |
| machines | NO | NO |
| machinesets | NO | NO |
| members | YES | YES |
| mutatingwebhookconfigurations | YES | YES |
| namespaces | NO | NO |
| networkinterfaces | YES | YES |
| networkpolicies | NO | NO |
| networks | NO | NO |
| nodes | NO | NO |
| oidcidentityproviders | YES | YES |
| orders | NO | NO |
| osimages | NO | NO |
| packageinstalls | NO | NO |
| packagemetadatas | YES | YES |
| packagerepositories | YES | YES |
| packages | NO | NO |
| persistentvolumeclaims | NO | NO |
| persistentvolumes | NO | NO |
| poddisruptionbudgets | NO | NO |
| pods | NO | NO |
| podsecuritypolicies | YES | YES |
| podtemplates | YES | YES |
| priorityclasses | YES | YES |
| prioritylevelconfigurations | YES | YES |
| projects | YES | YES |
| providerserviceaccounts | YES | YES |
| registries | YES | YES |
| replicasets | NO | NO |
| replicationcontrollers | NO | NO |
| resourcequotas | NO | NO |
| rolebindings* | NO | YES |
| roles* | NO | YES |
| runtimeclasses | YES | YES |
| secrets* | NO | YES |
| serviceaccounts | NO | NO |
| services | NO | NO |
| statefulsets | NO | NO |
| storageclasses | NO | NO |
| storagepolicies | YES | YES |
| supervisorservicedefinitions | NO | NO |
| supervisorservices | NO | NO |
| tanzukubernetesaddons | NO | NO |
| tanzukubernetesclusters | NO | NO |
| tanzukubernetesreleases | NO | NO |
| tcproutes | YES | YES |
| tkgserviceconfigurations | NO | NO |
| tokencredentialrequests | YES | YES |
| validatingwebhookconfigurations | YES | YES |
| vcuiplugins | YES | YES |
| virtualmachineclassbindings | NO | NO |
| virtualmachineclasses | NO | NO |
| virtualmachineimages | NO | NO |
| virtualmachines | NO | NO |
| virtualmachineservices | NO | NO |
| virtualmachinesetresourcepolicies | NO | NO |
| vmxnet3networkinterfaces | YES | YES |
| volumeattachments | YES | YES |
| vsphereclusters | NO | NO |
| vsphereclustertemplates | YES | YES |
| vspherecpiconfigs | NO | NO |
| vspherecsiconfigs | NO | NO |
| vspheredistributednetworks | YES | YES |
| vspheremachines | NO | NO |
| vspheremachinetemplates | NO | NO |
| vspherezones | NO | NO |
| wcpclusters | NO | NO |
| wcpmachines | NO | NO |
| wcpmachinetemplates | NO | NO |
| wcpnamespaces | YES | YES |
| webconsolerequests* | NO | YES |
| webhookauthenticators | YES | YES |
| whoamirequests | YES | YES |
There are a few resource, such as roles, rolebindings, secrets and webconsolerequests, which are query-able from a vSphere Namespace scope, but is not query-able from the Supervisor Default Namespace. All other resources behave the same in both the SV Default Namespace and a vSphere Namespace.
Note that some of the resource above may not be configured on your Supervisor, so a kubectl get may simply say the server does not have these resource types. However, it should not fail with the forbidden error.
There are other superset objects, one being cluster-api, which try to return a number of different objects. In the case of cluster-api, it tries to query objects such as clusterresourcesets and machinepools. Since these are forbidden, this query also fails.
If you do need to query these objects for any reason, you can SSH to one of the Supervisor Control Plane nodes. Run the script /usr/lib/vmware-wcp/decryptK8Pwd.py located on the vCenter Server that is managing the vSphere with Tanzu environment. This will allow you to gain SSH access to the Supervisor, and from there, all of the resources should be available to the kubectl get commands.
