Network Policies in Tanzu Mission Control revisited

Earlier this month, I had my first look at network policies in Tanzu Mission Control (TMC). This earlier post looked at a very simple network policy where I used a web server app, and showed how we could control access to it from other pods by using labels. In this post, I wanted to do something that is a bit more detailed. For the purposes of this test, I will use a pod based NFS server, and then control access to it from other pods who wish to mount the NFS file share from the server pod. I have already created the workspace in TMC, and attached the namespace where the NFS server and client pods are to be deployed. All of the details on how to do this, including the creation of a policy, are in the previous post so please refer to that post for those instructions. In this post, we will focus on how a network policy can control the communication between the NFS client and NFS server pods.

The yaml manifests used for the NFS server and client pods are available here, along with manifest for creating the server service, and client PVC and PV. I initially deployed this app in the namespace nfs-testing without any policy configured, and verified that I could successfully mount the NFS share exported from the NFS server pod on all of the NFS client pods. I was also able to do other communication between the NFS server and client pods such as ping requests. I then deleted the NFS client pods and just left the NFS server pod in place. Note the endpoints. This is where requests to the service via ClusterIP (10.69.133.143) are routed by kube-proxy. Thus, when we request an NFS shared to be mounted from the service IP address, it is routed to the NFS server pod on IP address 10.96.1.101.

% kubectl get pods -n nfs-testing -o wide -L app
NAME           READY   STATUS    RESTARTS   AGE    IP             NODE                                              NOMINATED NODE   READINESS GATES   APP
nfs-server-0   1       Running   0          157m   100.96.1.214   workload2-md-0-859f9d5496-zjlzj.corinternal.com   <none>           <none>            nfssrvr

% kubectl get svc -n nfs-testing
NAME        TYPE        CLUSTER-IP      EXTERNAL-IP  PORT(S)                      AGE
nfs-server  ClusterIP  100.69.133.143  <none>        2049/TCP,20048/TCP,111/TCP  12m

% kubectl get endpoints -n nfs-testing
NAME        ENDPOINTS                                              AGE
nfs-server  100.96.1.101:20048,100.96.1.101:111,100.96.1.101:2049  13m

Create a custom ingress policy in TMC

As mentioned, I wish to control who can access the NFS server pod. To do that I will create a custom ingress policy, and so that this policy only applies to my NFS server pod, I will add a pod selector which matches my NFS server pod (nfssrvr). A final step is to add a rule about who can access my NFS server pod ingress, i.e. the NFS client pods. I only want NFS client pods to access the ingress of the NFS server pod, basically pods in the 10.96.0.0/16 address range which is the CIDR that is shared by both NFS client and server pods. I will now login onto TMC, and create the rule with those requirements.

After creating the policy in TMC, it becomes visible in the nfs-testing namespace. Note the PodSelector and the CIDR which match what was added to the policy above.

% kubectl describe networkpolicy -n nfs-testing
Name:         tmc.wsp.nfs-testing.control-nfs-ingress
Namespace:    nfs-testing
Created on:   2021-11-16 15:14:30 +0000 GMT
Labels:       tmc.cloud.vmware.com/managed=true
Annotations:  <none>
Spec:
  PodSelector:     app=nfssrvr
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      IPBlock:
        CIDR: 100.96.0.0/16
        Except:
  Not affecting egress traffic
  Policy Types: Ingress

This policy should continue to allow the NFS clients to successfully start up, since this policy allows pod to pod communication between the NFS client and NFS server.

% kubectl get pods -o wide -n nfs-testing -L app -w
NAME               READY   STATUS              RESTARTS   AGE   IP             NODE                                              NOMINATED NODE   READINESS GATES   APP
nfs-client-pod-1   0/1     ContainerCreating   0          3s                   workload2-md-0-859f9d5496-djgsr.corinternal.com                                      nfs1
nfs-client-pod-2   0/1     ContainerCreating   0          2s                   workload2-md-0-859f9d5496-zjlzj.corinternal.com                                      nfs2
nfs-client-pod-3   0/1     ContainerCreating   0          2s                   workload2-md-0-859f9d5496-djgsr.corinternal.com                                      nfs3
nfs-server-0       1/1     Running             0          60m   100.96.1.214   workload2-md-0-859f9d5496-zjlzj.corinternal.com                                      nfssrvr
nfs-client-pod-1   1/1     Running             0          5s    100.96.2.193   workload2-md-0-859f9d5496-djgsr.corinternal.com                                      nfs1
nfs-client-pod-2   1/1     Running             0          5s    100.96.1.17    workload2-md-0-859f9d5496-zjlzj.corinternal.com                                      nfs2
nfs-client-pod-3   1/1     Running             0          5s    100.96.2.194   workload2-md-0-859f9d5496-djgsr.corinternal.com                                      nfs3

The NFS client pods appear to have started successfully. Let’s see if they have successfully mounted the NFS share (onto the /nfs directory mount-point in the pod).

% kubectl exec -it nfs-client-pod-1 -n nfs-testing -- sh
/ # df /nfs
Filesystem           1K-blocks      Used Available Use% Mounted on
100.69.133.143:/exports
                       5095424     20480   4796416   0% /nfs
/ # cd /nfs
/nfs # echo "hello-world" > hello
/nfs # cat hello
hello-world
/nfs #

Looks like everything is working as expected, and the NFS clients are still able to mount the NFS share from the NFS server. Let’s do some more tests on the policy.

Add an Exclude IP range

From the pod listing in the previous step, we can see that the NFS client pods came up on two different ranges – 10.96.1.0 and 10.96.2.0. As a test, let’s delete the client pods, add an Exclude IP range to the rule and set it to 100.96.2.0/24. Now let’s see what happens when we try to deploy the NFS client pods once more.

Here is the updated policy with the excluded IP address range added.

Let’s examine the policy from the TKG cluster perspective.

% kubectl describe networkpolicy -n nfs-testing
Name:         tmc.wsp.nfs-testing.control-nfs-ingress
Namespace:    nfs-testing
Created on:   2021-11-16 15:14:30 +0000 GMT
Labels:       tmc.cloud.vmware.com/managed=true
Annotations:  <none>
Spec:
  PodSelector:     app=nfssrvr
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      IPBlock:
        CIDR: 100.96.0.0/16
        Except: 100.96.2.0/24
  Not affecting egress traffic
  Policy Types: Ingress

Now when the NFS client pods are deployed, we can see that only the NFS client pod on the 100.96.1.0/24 network come online, and the two NFS client pods on the 100.96.2.0/24 network are stuck in ContainerCreating. The reason they are stuck is that they no longer have access to the NFS server pod to mount the share due to that IP range being excluded in the rule.

% kubectl apply  -f nfs-client-pod-1.yaml -f nfs-client-pod-2.yaml -f nfs-client-pod-3.yaml
pod/nfs-client-pod-1 created
pod/nfs-client-pod-2 created
pod/nfs-client-pod-3 created

% kubectl get pods -n nfs-testing -o wide -L app
NAME               READY   STATUS              RESTARTS   AGE    IP             NODE                                              NOMINATED NODE   READINESS GATES   APP
nfs-client-pod-1   0/1     ContainerCreating   0          16s             workload2-md-0-859f9d5496-djgsr.corinternal.com                          nfs1
nfs-client-pod-2   1/1     Running             0          16s    100.96.1.127   workload2-md-0-859f9d5496-zjlzj.corinternal.com                          nfs2
nfs-client-pod-3   0/1     ContainerCreating   0          15s             workload2-md-0-859f9d5496-djgsr.corinternal.com                          nfs3
nfs-server-0       1/1     Running             0          174m   100.96.1.214   workload2-md-0-859f9d5496-zjlzj.corinternal.com                          nfssrvr
 

After approximately 2 minutes, the NFS mount attempt should time out, and the following events will be observable on the pod.

% kubectl get events -A | grep nfs-client-pod-3
nfs-testing         2m14s       Normal    Scheduled                         pod/nfs-client-pod-3   \
            Successfully assigned nfs-testing/nfs-client-pod-3 to workload2-md-0-859f9d5496-zjlzj.corinternal.com
nfs-testing         11s         Warning   FailedMount                       pod/nfs-client-pod-3   \
            Unable to attach or mount volumes: unmounted volumes=[nfs-vol], unattached volumes=[default-token-xpmgh nfs-vol]: timed out waiting for the condition

We can go back and modify the policy so that it allows pods on the 100.96.2.0/24 network to access the ingress of the NFS server pod. Simply changing the exclude range from 100.96.2.0/24 to 100.96.3.0/24 will achieve this.

% kubectl describe networkpolicy -n nfs-testing
Name:         tmc.wsp.nfs-testing.control-nfs-ingress
Namespace:    nfs-testing
Created on:   2021-11-16 15:14:30 +0000 GMT
Labels:       tmc.cloud.vmware.com/managed=true
Annotations:  <none>
Spec:
  PodSelector:     app=nfssrvr
  Allowing ingress traffic:
    To Port: <any> (traffic allowed to all ports)
    From:
      IPBlock:
        CIDR: 100.96.0.0/16
        Except: 100.96.3.0/24
  Not affecting egress traffic
  Policy Types: Ingress


% kubectl get pods -o wide -n nfs-testing -L app
NAME               READY   STATUS    RESTARTS   AGE     IP             NODE                                              NOMINATED NODE   READINESS GATES   APP
nfs-client-pod-1   1/1     Running   0          3m45s   100.96.2.197   workload2-md-0-859f9d5496-djgsr.corinternal.com   <none>           <none>            nfs1
nfs-client-pod-2   1/1     Running   0          3m44s   100.96.2.198   workload2-md-0-859f9d5496-djgsr.corinternal.com   <none>           <none>            nfs2
nfs-client-pod-3   1/1     Running   0          3m43s   100.96.1.24    workload2-md-0-859f9d5496-zjlzj.corinternal.com   <none>           <none>            nfs3
nfs-server-0       1/1     Running   0          68m     100.96.1.214   workload2-md-0-859f9d5496-zjlzj.corinternal.com   <none>           <none>            nfssrvr

Add Port Rules

With the present policy, we are allowing not only NFS port access to the NFS server pod ingress, but in fact all ports are open on the NFS server pod from the NFS client pods. Now, we will add a rule that controls which ports are accessible on the NFS server (ingress) pod. Without any rules specified, all ports are available. A very simple example would be to show that ping is currently available between the client and the server (and vice-versa).

% kubectl exec -it nfs-client-pod-1 -n nfs-testing -- sh
/ # ping 100.96.1.214
PING 100.96.1.214 (100.96.1.214): 56 data bytes
64 bytes from 100.96.1.214: seq=0 ttl=62 time=0.982 ms
64 bytes from 100.96.1.214: seq=1 ttl=62 time=0.607 ms
64 bytes from 100.96.1.214: seq=2 ttl=62 time=0.271 ms
^C
--- 100.96.1.214 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.271/0.620/0.982 ms
/ # exit

 
% kubectl exec -it nfs-server-0 -n nfs-testing -- sh
sh-4.2# ping 100.96.2.218
PING 100.96.2.218 (100.96.2.218) 56(84) bytes of data.
64 bytes from 100.96.2.218: icmp_seq=1 ttl=62 time=0.962 ms
64 bytes from 100.96.2.218: icmp_seq=2 ttl=62 time=0.417 ms
64 bytes from 100.96.2.218: icmp_seq=3 ttl=62 time=0.278 ms
^C
--- 100.96.2.218 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2014ms
rtt min/avg/max/mdev = 0.278/0.552/0.962/0.295 ms
sh-4.2#

Let’s now go ahead and only allow ingress on the NFS specific ports over TCP, namely port 111, 2049 and 20048. The ports required for NFS are defined in the NFS server service, and can be queried as follows.

% kubectl get svc nfs-server -n nfs-testing
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
nfs-server   ClusterIP      100.69.133.143  <none>        2049:30331/TCP,20048:31149/TCP,111:31083/TCP   3h7m

By way of testing, let’s add access to only one of those ports initially. Let’s use port 111.

Let’s check the policy in the namespace:

% kubectl describe networkpolicy -n nfs-testing
Name:         tmc.wsp.nfs-testing.control-nfs-ingress
Namespace:    nfs-testing
Created on:   2021-11-16 15:14:30 +0000 GMT
Labels:       tmc.cloud.vmware.com/managed=true
Annotations:  <none>
Spec:
  PodSelector:     app=nfssrvr
  Allowing ingress traffic:
    To Port: 111/TCP
    From:
      IPBlock:
        CIDR: 100.96.0.0/16
        Except: 100.96.3.0/24
  Not affecting egress traffic
  Policy Types: Ingress

With this policy in place, all of the NFS client pods get stuck on ContainerCreating state as they are unable to mount the NFS file share from the NFS server. We need to include all 3 of the ports in the rules for it to work.

% kubectl describe networkpolicy -n nfs-testing
Name:         tmc.wsp.nfs-testing.control-nfs-ingress
Namespace:    nfs-testing
Created on:   2021-11-16 15:14:30 +0000 GMT
Labels:       tmc.cloud.vmware.com/managed=true
Annotations:  <none>
Spec:
  PodSelector:     app=nfssrvr
  Allowing ingress traffic:
    To Port: 111/TCP
    To Port: 2049/TCP
    To Port: 20048/TCP
    From:
      IPBlock:
        CIDR: 100.96.0.0/16
        Except: 100.96.3.0/24
  Not affecting egress traffic
  Policy Types: Ingress

And if we delete and recreate the NFS client pods with the port rules in place, we should hopefully see the pods start successfully.

% kubectl get pods -o wide -n nfs-testing -L app  -w
NAME               READY   STATUS    RESTARTS   AGE     IP             NODE                                              NOMINATED NODE   READINESS GATES   APP
nfs-client-pod-1   1/1     Running   0          2m25s   100.96.2.200   workload2-md-0-859f9d5496-djgsr.corinternal.com   <none>           <none>            nfs1
nfs-client-pod-2   1/1     Running   0          2m24s   100.96.2.201   workload2-md-0-859f9d5496-djgsr.corinternal.com   <none>           <none>            nfs2
nfs-client-pod-3   1/1     Running   0          2m23s   100.96.1.31    workload2-md-0-859f9d5496-zjlzj.corinternal.com   <none>           <none>            nfs3
nfs-server-0       1/1     Running   0          75m     100.96.1.214   workload2-md-0-859f9d5496-zjlzj.corinternal.com   <none>           <none>            nfssrvr

Success! All the pods have come online. This should mean that they are successfully able to mount the NFS share from the NFS server. Let’s login to one of the clients and make sure.

% kubectl exec -it nfs-client-pod-1 -n nfs-testing -- sh
/ # df /nfs
Filesystem           1K-blocks      Used Available Use% Mounted on
10.69.133.143:/exports
                       5095424     20480   4796416   0% /nfs
/ # ls /nfs
hello       lost+found
/ # cat /nfs/hello
hello-world
/ # echo "hello-again" >> /nfs/hello-again
/ # cat  /nfs/hello-again
hello-again
/ #

It looks like everything is working as expected. Now the NFS server should only be allowing ingress on the NFS ports, and nothing else. Let’s try a quick ping from a client to the server to verify.

/ # ping 100.96.1.214
PING 100.96.1.214 (100.96.1.214): 56 data bytes
^C
--- 100.96.1.214 ping statistics ---
19 packets transmitted, 0 packets received, 100% packet loss

Pings are now blocked between the NFS client and the NFS server. However, the ingress network policy has only been implemented on the NFS server, not on the NFS clients, since we used the PodSelector in the policy to specify that. Thus, it should still be possible to ping from the server to any of the clients. Let’s try:

% kubectl exec -it nfs-server-0 -n nfs-testing -- sh
sh-4.2# ping 100.96.2.220
PING 100.96.2.220 (100.96.2.220) 56(84) bytes of data.
64 bytes from 100.96.2.220: icmp_seq=1 ttl=62 time=0.924 ms
64 bytes from 100.96.2.220: icmp_seq=2 ttl=62 time=0.384 ms
64 bytes from 100.96.2.220: icmp_seq=3 ttl=62 time=0.271 ms
64 bytes from 100.96.2.220: icmp_seq=4 ttl=62 time=0.243 ms
^C
--- 100.96.2.220 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.243/0.455/0.924/0.276 ms
sh-4.2#

Yes – that still works since it is not blocked by any network rules. Everything is working as expected.

Tips

Note: When creating rules, TMC won’t report directly that it failed to create it. All that will happen is that the rule with not appear in the namespace as expected. If this happens, click on the Policies > Insights view, and check the network for issues. If there is a problem with the policy, it will be reported here:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.