Why do I get “Error from server (Forbidden)” in vSphere with Tanzu

I’ve seen a number of queries around the behaviour of vSphere with Tanzu when it comes to querying Kubernetes objects on the Supervisor Cluster. More often than not, it is a question which arises when a user get an error similar to the following:

Error from server (Forbidden): wcpnamespaces.appplatform.wcp.vmware.com is forbidden: \
User "sso:Administrator@vsphere.local" cannot list resource "wcpnamespaces" in API group \
"appplatform.wcp.vmware.com" in the namespace "cormac-ns"

The reason for these errors is because the Supervisor Cluster is not treated as a general purpose Kubernetes cluster. The predominant role of the Supervisor Cluster is to provide services, such as the TKG Service for provisioning Kubernetes clusters and the VM Service for provisioning virtual machines. However, when troubleshooting the Supervisor cluster, errors like the one shown above may make you think something is wrong and may set you off on the wrong direction. Therefore, I provide a list of all API resources from my vSphere 8.0 environment below. I highlight which ones are query-able from the kubectl command when running in the Supervisor Cluster Default Namespace context, and which are query-able when running in a user-created vSphere Namespace context. This will hopefully help allay any fears that something is not correct on the Supervisor. The first column is the name of API resource, retrieved from the command kubectl api-resources. This has been sorted in alphabetical order, and only the name of the resource is displayed. The second column shows whether or not a kubectl get can be run against this resource from a vSphere Namespace context or if it is forbidden. The last column is a kubectl get run against the Supervisor Default Namespace, i.e. when the context is set to the Supervisor Cluster API Server, and again shows whether or not it is forbidden. Note that these outputs are mostly the same, but there are a handful of differences.

API Resources Forbidden? (vSphere Namespace context) Forbidden? (SV Default Namespace context)
activedirectoryidentityproviders YES YES
agentinstalls NO NO
antreaconfigs NO NO
apiservices YES YES
apps YES YES
availabilityzones YES YES
aviloadbalancerconfigs YES YES
blockaffinities YES YES
calicoconfigs NO NO
capabilities YES YES
ccsplugins NO NO
certificaterequests NO NO
certificates NO NO
certificatesigningrequests YES YES
challenges NO NO
cliplugins NO NO
clusterbootstraps NO NO
clusterbootstraptemplates NO NO
clusterclasses NO NO
clusterissuers YES YES
clusterresourcesetbindings YES YES
clusterresourcesets YES YES
clusterrolebindings YES YES
clusterroles YES YES
clusters NO NO
cnscsisvfeaturestates YES YES
cnsnodevmattachments NO NO
cnsvolumemetadatas YES YES
cnsvolumeoperationrequests YES YES
compatibilities YES YES
componentstatuses YES YES
configmaps NO NO
contentlibraryproviders NO NO
contentsourcebindings NO NO
contentsources NO NO
controllerrevisions NO NO
credentialissuers YES YES
cronjobs NO NO
csidrivers YES YES
csinodes YES YES
csistoragecapacities YES YES
customresourcedefinitions YES YES
daemonsets NO NO
deployments NO NO
endpoints NO NO
endpointslices NO NO
events NO NO
featuregates NO NO
features NO NO
federationdomains YES YES
flowschemas YES YES
gatewayclasses YES YES
gateways NO NO
haproxyloadbalancerconfigs YES YES
horizontalpodautoscalers NO NO
httproutes YES YES
imagedisks NO NO
images NO NO
ingressclasses YES YES
ingresses NO NO
internalpackagemetadatas YES YES
internalpackages YES YES
ipamblocks YES YES
ipamconfigs YES YES
ipamhandles YES YES
ippools YES YES
issuers NO NO
jobs NO NO
jwtauthenticators YES YES
kappcontrollerconfigs NO NO
kubeadmconfigs NO NO
kubeadmconfigtemplates NO NO
kubeadmcontrolplanes NO NO
kubeadmcontrolplanetemplates YES YES
kuberneteslicenses NO NO
ldapidentityproviders YES YES
leases YES YES
limitranges NO NO
loadbalancerconfigs YES YES
machinedeployments NO NO
machinehealthchecks NO NO
machinepools YES YES
machines NO NO
machinesets NO NO
members YES YES
mutatingwebhookconfigurations YES YES
namespaces NO NO
networkinterfaces YES YES
networkpolicies NO NO
networks NO NO
nodes NO NO
oidcidentityproviders YES YES
orders NO NO
osimages NO NO
packageinstalls NO NO
packagemetadatas YES YES
packagerepositories YES YES
packages NO NO
persistentvolumeclaims NO NO
persistentvolumes NO NO
poddisruptionbudgets NO NO
pods NO NO
podsecuritypolicies YES YES
podtemplates YES YES
priorityclasses YES YES
prioritylevelconfigurations YES YES
projects YES YES
providerserviceaccounts YES YES
registries YES YES
replicasets NO NO
replicationcontrollers NO NO
resourcequotas NO NO
rolebindings* NO YES
roles* NO YES
runtimeclasses YES YES
secrets* NO YES
serviceaccounts NO NO
services NO NO
statefulsets NO NO
storageclasses NO NO
storagepolicies YES YES
supervisorservicedefinitions NO NO
supervisorservices NO NO
tanzukubernetesaddons NO NO
tanzukubernetesclusters NO NO
tanzukubernetesreleases NO NO
tcproutes YES YES
tkgserviceconfigurations NO NO
tokencredentialrequests YES YES
validatingwebhookconfigurations YES YES
vcuiplugins YES YES
virtualmachineclassbindings NO NO
virtualmachineclasses NO NO
virtualmachineimages NO NO
virtualmachines NO NO
virtualmachineservices NO NO
virtualmachinesetresourcepolicies NO NO
vmxnet3networkinterfaces YES YES
volumeattachments YES YES
vsphereclusters NO NO
vsphereclustertemplates YES YES
vspherecpiconfigs NO NO
vspherecsiconfigs NO NO
vspheredistributednetworks YES YES
vspheremachines NO NO
vspheremachinetemplates NO NO
vspherezones NO NO
wcpclusters NO NO
wcpmachines NO NO
wcpmachinetemplates NO NO
wcpnamespaces YES YES
webconsolerequests* NO YES
webhookauthenticators YES YES
whoamirequests YES YES

There are a few resource, such as roles, rolebindings, secrets and webconsolerequests, which are query-able from a vSphere Namespace scope, but is not query-able from the Supervisor Default Namespace. All other resources behave the same in both the SV Default Namespace and a vSphere Namespace.

Note that some of the resource above may not be configured on your Supervisor, so a kubectl get may simply say the server does not have these resource types. However, it should not fail with the forbidden error.

There are other superset objects, one being cluster-api, which try to return a number of different objects. In the case of cluster-api, it tries to query objects such as clusterresourcesets and machinepools. Since these are forbidden, this query also fails.

If you do need to query these objects for any reason, you can SSH to one of the Supervisor Control Plane nodes. Run the script /usr/lib/vmware-wcp/decryptK8Pwd.py located on the vCenter Server that is managing the vSphere with Tanzu environment. This will allow you to gain SSH access to the Supervisor, and from there, all of the resources should be available to the kubectl get commands.