A first look at the Couchbase Operator

A few weeks back, I took a look at Heptio Velero, formely known as Ark. Velero provides backup and restore capabilities for cloud native applications. During that research, I used a Couchbase DB as my application of choice for backup/restore. After speaking to Couchbase regarding that blog post, they strongly recommended I try the new Couchbase operator rather than the StatefulSet method that I was using for the application. Couchbase talk about the advantages of the operator approach over StatefulSets here.

Now, while Couchbase provide steps on how to deploy Couchbase with their operator, they create it in the default K8s namespace. In my test, I want to put Couchbase in its own namespace. The steps provided here are provided to get you started with the new Couchbase Operator, running on vSphere and vSAN infrastructure, in its own Kubernetes namespace. I also talk about some issues with the bundled load generating tool, called pillowfight.

Couchbase provide prescriptive instructions on how to get started with their operator here. It includes all the necessary configuration files. A few things about the operator:

  • When loaded, it downloads the Operator Docker image as specified in the operator.yaml file. It uses a deployment construct so that it can restart if the pod it is running in dies.
  • It creates the CouchbaseCluster custom resource definition (CRD)
  • It starts listening for CouchbaseCluster events.

I made a few modifications to allow Couchbase to run in its own namespace:

  • I first of all created a new namespace (obviously) called couchbase.
  • When the cluster role was created, I created the service account in the new couchbase namespace and I then assigned the cluster role to that service account using a cluster role binding.
  • I modified the operator.yaml file to include a metadata.namespace=couchbase setting so that it applies to the couchbase namespace

By monitoring the logs of the couchbase operator pod, we can observe the following startup messages:

$ kubectl logs couchbase-operator-6fcfbd8599-zqsh2 -n couchbase
time="2019-03-20T09:27:41Z" level=info msg="couchbase-operator v1.1.0 (release)" module=main
time="2019-03-20T09:27:41Z" level=info msg="Obtaining resource lock" module=main
time="2019-03-20T09:27:41Z" level=info msg="Starting event recorder" module=main
time="2019-03-20T09:27:41Z" level=info msg="Attempting to be elected the couchbase-operator leader" module=main
time="2019-03-20T09:27:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Endpoints\", Namespace:\"couchbase\", Name:\"couchbase-operator\", UID:\"68fece18-4af2-11e9-be9b-005056a24d92\", APIVersion:\"v1\", ResourceVersion:\"1596774\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' couchbase-operator-6fcfbd8599-zqsh2 became leader" module=event_recorder
time="2019-03-20T09:27:41Z" level=info msg="I'm the leader, attempt to start the operator" module=main
time="2019-03-20T09:27:41Z" level=info msg="Creating the couchbase-operator controller" module=main
time="2019-03-20T09:27:46Z" level=info msg="CRD initialized, listening for events..." module=controller
time="2019-03-20T09:27:46Z" level=info msg="starting couchbaseclusters controller"

I was now ready to deploy the Couchbase cluster using the new cbopctl CLI tool. I also needed to make a few changes to the default cluster configuration file (couchbase-cluster-sc.yaml).

  • I placed it in the couchbase namespace with the  metadata.namespace entry
  • I set spec.disableBucketManagement to true which allows me to make changes to buckets via UI/CLI (otherwise I have to make all changes via edits to the YAML file)
  • I added Persistent Volumes for the default and data mounts (I had to create a new StorageClass for the volumeClaimTemplate to use for this – see below)

Here is the full CouchbaseCluster YAML file with my changes highlighted.

apiVersion: couchbase.com/v1
kind: CouchbaseCluster
metadata:
  name: cb-example
  namespace: couchbase
spec:
  securityContext:
    fsGroup: 1000
  baseImage: couchbase/server
  version: enterprise-5.5.2
  authSecret: cb-example-auth
  exposeAdminConsole: true
  disableBucketManagement: true
  adminConsoleServices:
    - data
  cluster:
    dataServiceMemoryQuota: 256
    indexServiceMemoryQuota: 256
    searchServiceMemoryQuota: 256
    eventingServiceMemoryQuota: 256
    analyticsServiceMemoryQuota: 1024
    indexStorageSetting: memory_optimized
    autoFailoverTimeout: 120
    autoFailoverMaxCount: 3
    autoFailoverOnDataDiskIssues: true
    autoFailoverOnDataDiskIssuesTimePeriod: 120
    autoFailoverServerGroup: false
  buckets:
    - name: default
      type: couchbase
      memoryQuota: 128
      replicas: 1
      ioPriority: high
      evictionPolicy: fullEviction
      conflictResolution: seqno
      enableFlush: true
      enableIndexReplica: false
  servers:
    - size: 3
      name: all_services
      services:
        - index
        - query
        - search
        - eventing
        - analytics
        - data
      pod:
        volumeMounts:
          default: couchbase
          data:  couchbase
  volumeClaimTemplates:
    - metadata:
        name: couchbase
      spec:
        storageClassName: "couchbasesc"
        resources:
          requests:
            storage: 1Gi

I am skipping over the authentication and user requirements which are all documented on the Couchbase site. However, once the application has been deployed, you should be able to see the following in the couchbase operator pod logs:

$ kubectl logs couchbase-operator-6fcfbd8599-zqsh2 -n couchbase
.
.
time="2019-03-20T09:48:34Z" level=info msg="Watching new cluster" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Janitor process starting" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Setting up client for operator communication with the cluster" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Cluster does not exist so the operator is attempting to create it" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Creating headless service for data nodes" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Creating NodePort UI service (cb-example-ui) for data nodes" cluster-name=cb-example module=cluster
time="2019-03-20T09:48:34Z" level=info msg="Creating a pod (cb-example-0000) running Couchbase enterprise-5.5.2" cluster-name=cb-example module=cluster

And if everything comes up successfully, you can query the pods, persistent volumes and services as they initialize.

$ kubectl get po -n couchbase
NAME                                  READY   STATUS    RESTARTS   AGE
cb-example-0000                       1/1     Running   0          7m8s
cb-example-0001                       1/1     Running   0          6m7s
cb-example-0002                       1/1     Running   0          5m14s
couchbase-operator-6fcfbd8599-zqsh2   1/1     Running   0          28m

$ kubectl get pv -n couchbase
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                STORAGECLASS   REASON   AGE
pvc-100e3c17-40e3-11e9-be9b-005056a24d92   10Gi       RWO            Delete           Bound    velero/minio-pv-claim-1                              minio-sc                12d
pvc-53c6962f-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0000-00-default   couchbasesc             7m51s
pvc-5b30f298-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0000-01-data      couchbasesc             7m44s
pvc-795bcf7b-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0001-00-default   couchbasesc             6m51s
pvc-7edc6a4d-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0001-01-data      couchbasesc             6m43s
pvc-97e9f6a9-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0002-00-data      couchbasesc             6m2s
pvc-9bcc3d4d-4af5-11e9-be9b-005056a24d92   1Gi        RWO            Delete           Bound    couchbase/pvc-couchbase-cb-example-0002-01-default   couchbasesc             5m50s

$ kubectl get svc -n couchbase
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
cb-example       ClusterIP   None                           8091/TCP,18091/TCP               8m55s
cb-example-srv   ClusterIP   None                           11210/TCP,11207/TCP              8m55s
cb-example-ui    NodePort    10.100.200.132   <none>        8091:32635/TCP,18091:30198/TCP   3m20s

So, to recap, the steps are:

  1. Follow the documented steps to set up the couchbase ClusterRole
  2. Create the couchbase namespace – kubectl create ns couchbase
  3. Create the couchbase operator service account in the couchbase namespace – kubectl create serviceaccount couchbase-operator –namespace couchbase
  4. Create the operator (modified for the couchbase namespace) – kubectl create -f operator.yaml
  5. Create the necessary secrets (modified for the couchbase namespac) – kubectl create -f secret.yaml
  6. Create the couchbase cluster using cbopctl – cbopctl create -f couchbase-cluster-sc.yaml

In the last output, the services, we have the port mappings for the Couchbase UI, both for http and https. If we connect to any of the K8s slave nodes with those ports, we should be able to access our Couchbase deployment, using the Administrator/password credential provided in our configuration.

Initially there are no buckets created. There is one defined in the cluster configuration but we overrode it with the advanced setting. We can quickly remedy that by creating a few temp ones. I will add two – the first is called default and the second is called cormac. At the moment there are no items in either bucket.

Couchbase also bundle a utility called pillowfight which is a very useful way of populating the buckets. For some reason, I had some issues with the bundled “sequioatools” version of pillowfight. Once I reverted to using the “couchbaseutils” version, all was good. You will need to set up the appropriated user credentials to do this, but once again this is all documented on the main Couchbase operator site. Here are my sample YAML files for populating the default bucket:

apiVersion: batch/v1
kind: Job
metadata:
  name: pillowfight
  namespace: couchbase
spec:
  template:
    metadata:
      name: pillowfight
    spec:
      containers:
      - name: pillowfight
        image: couchbaseutils/pillowfight:v2.9.3
        command: ["cbc-pillowfight",
                  "-U", "couchbase://cb-example-0000.cb-example.couchbase.svc.cluster.local/default?select_bucket=true",
                  "-I", "10000", "-B", "1000", "-c", "10", "-t", "1", "-P", "password"]
      restartPolicy: Never

The only difference for the cormac bucket is the command syntax is slightly different:

command: ["cbc-pillowfight",
          "-U", "couchbase://cb-example-0000.cb-example.couchbase.svc.cluster.local/cormac",
          "-I", "10000", "-B", "1000", "-c", "10", "-t", "1", "-u", "Administrator", "-P", "password"]

After running the pillowfight jobs, you will see new pods that have already completed their tasks:

$ kubectl get po -n couchbase
NAME                                  READY   STATUS      RESTARTS   AGE
cb-example-0000                       1/1     Running     0          20m
cb-example-0001                       1/1     Running     0          19m
cb-example-0002                       1/1     Running     0          18m
couchbase-operator-6fcfbd8599-ggv98   1/1     Running     0          24m
create-user-dk6xg                     0/1     Completed   0          89s
pillowfight-fqvgp                     0/1     Completed   0          70s
pillowfightcormac-dmqnf               0/1     Completed   0          7s

More importantly, if we take a look at the Couchbase UI, we see that we now have 1,000 items in each bucket:

And that’s it. You are up and running with your Couchbase operator. And to close, this was provisioned in a K8s cluster on top of VMware PKS, vSphere and vSAN infrastructure. BTW, the issue with pillowfight was reported here.