TKG v1.4 Prometheus + Grafana Package Deployment: package reconciliation failed

by Posted on

I was recently running through the exercise of deploying Cert Manager, Contour (+ Envoy Ingress), Prometheus and Grafana packages available with TKG v1.4, just to see what steps were involved in setting up a full monitoring stack for my TKG cluster. This was a TKG deployment to vSphere, using the NSX Advanced Load Balancer for Load Balancer functionality. You can read about the new enhancements around the NSX ALB and TKG v1.4 here.  Honestly, it is pretty straight-forward, with some detailed documentation on the topic available here. Everything was plain sailing until I tried to deploy the Grafana package with, for the most part, default settings. I was using this very simplified values file with the deployment:

ingress:
  virtual_host_fqdn: "grafana.corinternal.com"

It was when I tried to install the Grafana package with that values file that I encountered the following error:

$ tanzu package install grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 --values-file grafana.yaml
- Installing package 'grafana.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-default-sa'
| Creating cluster admin role 'grafana-default-cluster-role'
| Creating cluster role binding 'grafana-default-cluster-rolebinding'
| Creating secret 'grafana-default-values'
- Creating package resource
/ Package install status: Reconciling

Please consider using 'tanzu package installed update' to update the installed package with correct settings

Error: package reconciliation failed: kapp: Error: Ownership errors:
- Resource 'namespace/tanzu-system-monitoring (v1) cluster' is already associated with a different app 'prometheus-ctrl' namespace: default (label 'kapp.k14s.io/app=1632306561798628761')
Usage:
  tanzu package install INSTALLED_PACKAGE_NAME --package-name PACKAGE_NAME --version VERSION [flags]
The reason for this is that Prometheus, which I had already installed as part of the monitoring stack, is using the same namespace that Grafana is also trying to use. Thus, we need to change the tanzu-system-monitoring namespace that Grafana uses to another namespace to avoid the issue. This request to change the namespace for Grafana to tanzu-system-dashboards is highlighted in the documentation – see Deploy Grafana on a Tanzu Kubernetes Cluster step 4, although you might skip over it in the belief that it is not a requirement. It is indeed a requirement if you wish to deploy both the Prometheus and Grafana packages to the same cluster. You might also ask why this is the default if this is an interoperability issue? Rest assured we are aware of it and this will be addressed in an upcoming version. So how do we change to a new namespace? This can be done through the values file that is provided to Grafana at deployment time, which we saw previously. Here again is an updated, but still very simplified, Grafana values file that I have used to workaround the issue.
namespace: tanzu-system-dashboard

ingress:
  virtual_host_fqdn: "grafana.corinternal.com"
And now the Grafana pack should install successfully:
$ tanzu package install grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 --values-file grafana.yaml
- Installing package 'grafana.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-default-sa'
| Creating cluster admin role 'grafana-default-cluster-role'
| Creating cluster role binding 'grafana-default-cluster-rolebinding'
| Creating secret 'grafana-default-values'
- Creating package resource
| Package install status: Reconciling
 
Added installed package 'grafana' in namespace 'default'

$ kubectl get pods -A | grep grafana
tanzu-system-dashboard    grafana-857cb96d5-hvjqv       2/2     Running   0          5m40s

$ kubectl get svc -A | grep grafana
tanzu-system-dashboard    grafana                   LoadBalancer   100.67.2.164     XX.XX.13.199   80:32313/TCP                 5m48s

And now if you connect to one of the available Grafana dashboards, it should be possible to see the Prometheus metrics scraped from the cluster displayed in the dashboard. This is because a Prometheus data source (taken from the Prometheus package) gets automatically added to this Grafana configuration. Note that there are two Grafana dashboards automatically included in the package.

Published by Cormac

2 Replies to “TKG v1.4 Prometheus + Grafana Package Deployment: package reconciliation failed”

Comments are closed.