Some nice enhancements to Docker Volume Driver for vSphere v0.7

dockerThis week I am over at our VMware HQ in Palo Alto. I caught up with the guys in our storage team who are working on our docker volume driver for vSphere to find out what enhancements they have made with version 0.7. They have added some cool new enhancements which I think you will like.

First, this has been designed specifically for docker version 1.12. So the first thing you will have to do is to make sure that your docker is at this latest version. For most distros, this is quite a simple thing to do. But since I predominantly use our Photon OS distro, which ships with docker version 1.11 currently, there are a few additional steps to consider. To update the version of docker on Photon OS, you can use the following steps:

Step 1. Make a backup of /etc/yum.repos.d/photon-dev.repo and then add the following stanza of text to the original:

[photon-dev]
name=VMware Photon Linux Dev(x86_64)
baseurl=https://dl.bintray.com/vmware/photon_dev_$basearch
gpgkey=file:///etc/pki/rpm-gpg/VMWARE-RPM-GPG-KEY
gpgcheck=1
enabled=1
skip_if_unavailable=True

Step 2. Run  the following command to update docker:

tdnf install --refresh docker

Step 3. Make a backup of /usr/lib/systemd/system/docker-containerd.service and the edit the original to modify the “ExecStart” section. Replace it with the following:

ExecStart=/usr/bin/docker-containerd --listen 
unix:///run/containerd.sock --runtime /usr/bin/docker-runc 
--shim /usr/bin/docker-containerd-shim
 Step 4. Run the following commands:
systemctl daemon-reload

systemctl restart docker

Step 5. Verify that the docker version is at v1.12 and is functioning:

docker –v

docker ps

docker volume ls 

OK. Now I can proceed with the installation of docker volume driver version 0.7.

The installation process is still the same as before. There are two components; the VIB and the RPM. You need to install the VIB on the ESXi host, and you need to install the Guest OS component as well. The binaries are available for download on github to get you started quickly. But you can also built it yourself. This time I pulled down the latest build from github to my Photon OS VM using “git clone”, and I built the components on my Photon OS. If you want to do the same on your Photon OS, here are the steps:

Step 1. Install git using the following command

tdnf install git

Step 2. Clone the docker volume driver repo

git clone https://github.com/vmware/docker-volume-vsphere.git

Step 3. Install the make utility

tdnf install make

Step 4. Make sure docker is running in Photon OS. The build steps uses docker.

systemctl start docker
systemctl enable docker

Step 5. Build the code

cd docker-volume-vsphere/
make

Step 5. Install the RPM on Photon OS

cd build
rpm -ivh docker-volume-vsphere-0.7.8d42baa-1.x86_64.rpm

Step 6. restart docker

systemctl restart docker

Step 7. Copy the VIB to the ESXi host on which your Photon OS resides, and install it. The following command is run from the ESXi host, not the Photon OS, after the VIB was copied from the build folder on my Photon OS guest to the /scratch/downloads on the ESXi host:

 esxcli software vib install -d \
/scratch/downloads/vmware-esx-vmdkops-0.7.8d42baa.zip \
--no-sig-check -f

Now we can look at some of the enhancements in this newer version. First, there are some new field in the volume view from the ESXi host. Here is how things used to look in the previous version of the driver:

docker-vol-drvr-v0-7This is how they look now, with the additional fields:

docker-volume-driver-v0-7-newThere are a few new fields such as filesystem type, access mode and attachment mode. I’ll discuss these in just a moment.

There are some nice new enhancements to the “docker volume create” commands from within the Guest OS too. There is now a much nicer and more comprehensive help output if you make a typo/mistake. For example:

root@photon [ ~ ]# docker volume create -d vmdk \
-o bad_opt=bad_val
Error response from daemon: create 
.....: 
Valid options and defaults: 
[('size', '100mb'), 
('vsan-policy-name', '[VSAN default]'), 
('diskformat', 'thin'), 
('attach_as', 'independent_persistent')]
root@photon [ ~ ]#

There are a few things to highlight here.

  1. You can now select which filesystem to place on the volume, so long as the Guest OS supports it. This defaults to “ext4”.
  2. You can create Thin, LazyZeroedThick or EagerZeroedThick volumes. This defaults to thin.
  3. You can specify a VSAN policy, which I highlighted in a previous post here. If you do not specify a policy when creating volumes on VSAN, it uses the default policy.
  4. You can attach volumes as independent persistent, meaning that they can now be snapshot’ed for the purposes of backup. More on disk formats can be found here. The default is independent_persistent, meaning that no snapshot is created during backups, meaning that this volume is not backed up.

So lets create some new volumes for containers from my Photon OS Guest OS. I will create 3 volumes in total, with different specifications.

Volume 1 – Create a volume that is on the same datastore as my Photon OS VM, with a size of 10GB (which is a VSAN datastore by the way). Everything is default.

root@photon [ / ]# docker volume create --driver=vmdk \
--name=testvol1 -o size=10gb

testvol1

Volume 2 – Create a volume on my NFS array with a size of 10GB, but make it read-only.

root@photon [ / ]# docker volume create --driver=vmdk \
--name=testvol2@isilion-nfs-01 -o size=10gb -o access=read-only

testvol2@isilion-nfs-01

Volume 3 – Create a volume on my local VMFS datastore, but make it persistent rather than independent-persistent, so I can snapshot it for backup purposes.

root@photon [ / ]# docker volume create --driver=vmdk \
--name=testvol3@esxi-hp-08-local -o size=10gb \
-o attach-as=persistent

testvol3@esxi-hp-08-local

Lets take a look at the docker volume listing:

root@photon [ / ]# docker volume ls
DRIVER              VOLUME NAME
.
.
vmdk                testvol1
vmdk                testvol2@isilion-nfs-01
vmdk                testvol3@esxi-hp-08-local
.
root@photon [ / ]#

Again, in this version of the docker volume driver, testvol1 is not shown with an “@datastore” as it is on the same datastore as the Guest OS. We can use the docker inspect command to show the attributes of a particular volume:

root@photon [ / ]# docker volume inspect testvol2@isilion-nfs-01
[
    {
        "Name": "testvol2@isilion-nfs-01",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/testvol2@isilion-nfs-01",
        "Status": {
            "access": "read-only",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "164MB",
                "size": "10GB"
            },
            "created": "Thu Oct  6 14:21:32 2016",
            "created by VM": "Photon-DVD4V",
            "datastore": "isilion-nfs-01",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

Finally we can get an overview of all the volumes from an ESXi host perspective.

vmdkops-lsThis is what happens if you attempt to create something on the read-only filesystem.

root@photon [ / ]# docker run -it --rm -v testvol2@isilion-nfs-01:/testvol2 busybox sh
/ # df
Filesystem           1K-blocks      Used Available Use% Mounted on
overlay                8122788   2657456   5029676  35% /
tmpfs                  1026644         0   1026644   0% /dev
tmpfs                  1026644         0   1026644   0% /sys/fs/cgroup
/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0
                      10190136     23028   9626436   0% /testvol2
/dev/root              8122788   2657456   5029676  35% /etc/resolv.conf
/dev/root              8122788   2657456   5029676  35% /etc/hostname
/dev/root              8122788   2657456   5029676  35% /etc/hosts
shm                      65536         0     65536   0% /dev/shm
tmpfs                  1026644         0   1026644   0% /proc/sched_debug
/ # cd /testvol2/
/testvol2 # ls
lost+found
/testvol2 # mkdir zzz
mkdir: can't create directory 'zzz': Read-only file system
/testvol2 #

So lots of really great improvements as you can see. If you want to build persistent storage around containers from VMs which are running on vSphere, the docker volume driver for vSphere is the perfect way to do it. You can get all the information you need from the github page – https://vmware.github.io/docker-volume-vsphere/ – and it is also the best place to get help/assistance, or indeed contribute to the project. The team would only be too happy to help.

If you are VMworld in Europe, check out the Storage for Cloud Native Applications session – STO7831 – with Mark Sterin. This takes place on Tuesday, Oct 18, 5:00 p.m. – 6:00 p.m. Mark will demonstrate a lot of the goodness I am talking about here, and some other cool storage/container activities taking place at VMware.