Container Volumes in VIC v0.4.0

I mentioned yesterday that VMware made vSphere Integrated Containers (VIC) v0.4.0 available. Included in this version is support for container volumes. Now, as mentioned yesterday, VIC is still a work in progress, and not everything has yet been implemented. In this post I want to step you through some of the enhancements that we have made around docker volume support in VIC. This will hopefully provide you with enough information so that you can try this out for yourself.

To begin with, you need to ensure that a “volume store” is created when the VCH (Virtual Container Host) is deployed. This is a datastore and folder where the volumes are stored, and for ease of use, you can apply a label to it. One useful tit-bit here is that if you use the label “default” for your volume store, you do not have to specify it in the docker volume command line. For example, if I deploy a VCH as follows, note the –volume-store parameter. The format is “label:datastore/folder-on-datastore”. Here I have requested that the volume store be placed on the isilion-nfs-01 datastore in the folder called docker-vols. I have also labeled it as “default”. For information on the other parameters, refer to my previous post – Getting started with VIC v0.4.0.

root@photon [ /workspace/vic ]# ./vic-machine-linux create  \
--bridge-network Bridge-DPG --image-datastore isilion-nfs-01 \
-t 'administrator@vsphere.local:zzzzz@'  \
--compute-resource Mgmt --external-network VMNW51 --name VCH01 \
--volume-store "default:isilion-nfs-01/docker-vols"
INFO[2016-07-14T11:25:13Z] ### Installing VCH ####
INFO[2016-07-14T11:25:13Z] Generating certificate/key pair - private key in ./VCH01-key.pem
INFO[2016-07-14T11:25:13Z] Validating supplied configuration
INFO[2016-07-14T11:25:13Z] Firewall status: DISABLED on /CNA-DC/host/Mgmt/
INFO[2016-07-14T11:25:13Z] Firewall configuration OK on hosts:
INFO[2016-07-14T11:25:13Z]   /CNA-DC/host/Mgmt/
INFO[2016-07-14T11:25:13Z] License check OK on hosts:
INFO[2016-07-14T11:25:13Z]   /CNA-DC/host/Mgmt/
INFO[2016-07-14T11:25:13Z] DRS check OK on:
INFO[2016-07-14T11:25:13Z]   /CNA-DC/host/Mgmt/Resources
INFO[2016-07-14T11:25:14Z] Creating Resource Pool VCH01
INFO[2016-07-14T11:25:14Z] Creating directory [isilion-nfs-01] docker-vols
INFO[2016-07-14T11:25:14Z] Datastore path is [isilion-nfs-01] docker-vols
INFO[2016-07-14T11:25:14Z] Creating appliance on target
INFO[2016-07-14T11:25:14Z] Network role client is sharing NIC with external
INFO[2016-07-14T11:25:14Z] Network role management is sharing NIC with external
INFO[2016-07-14T11:25:15Z] Uploading images for container
INFO[2016-07-14T11:25:15Z]      bootstrap.iso
INFO[2016-07-14T11:25:15Z]      appliance.iso
INFO[2016-07-14T11:25:20Z] Registering VCH as a vSphere extension
INFO[2016-07-14T11:25:26Z] Waiting for IP information
INFO[2016-07-14T11:25:52Z] Waiting for major appliance components to launch
INFO[2016-07-14T11:25:52Z] Initialization of appliance successful
INFO[2016-07-14T11:25:52Z] Log server:
INFO[2016-07-14T11:25:52Z] DOCKER_HOST=
INFO[2016-07-14T11:25:52Z] Connect to docker:
INFO[2016-07-14T11:25:52Z] docker -H --tls info
INFO[2016-07-14T11:25:52Z] Installer completed successfully
root@photon [ /workspace/vic ]#

Now that my VCH is deployed and my docker endpoint is available, we can use the docker command to create volumes and attach them to containers.

First thing to note – the “docker volume ls” and the “docker volume inspect” commands are not yet implemented. So we do not yet have a good way of examining the storage consumption and layout through the docker API. This is work in progress however. That aside, we can still create and consume volumes. Here is how to do that.

root@photon [ /workspace/vic ]#  docker -H --tls volume create \
--name=demo --opt Capacity=1024
root@photon [ /workspace/vic ]#
Notice that I did not specify which “volume store” to use; it simply defaulted to “default” which is the docker-vols folder on my isilion-nfs-01 datastore. Let’s now take a look and see what got created from a vSphere perspective. If I select the datastore in the inventory, then view the files, I see a <volume-name>.VMDK was created in the folder docker-vols/VIC/volumes/<volume-name>:

files viewSo one thing to point out here – the Capacity=1024 option in the docker volume create command is a representation of 1MB blocks. So what was created is a 1GB VMDK. My understanding is that we will add additional granularity to this capacity option going forward.

Now to create a container to consume this volume. Let’s start with an Ubuntu image:

root@photon [ /workspace/vic ]# docker -H --tls run \
-v demo:/demo -it ubuntu /bin/bash

root@3ef0b682bc8d:/# mount | grep demo
/dev/sdb on /demo type ext4 (rw,noatime,data=ordered)

root@3ef0b682bc8d:/# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 965M 0 965M 0% /dev
tmpfs 1003M 0 1003M 0% /dev/shm
tmpfs 1003M 136K 1003M 1% /run
tmpfs 1003M 0 1003M 0% /sys/fs/cgroup
/dev/sda 7.8G 154M 7.2G 3% /
tmpfs 128M 43M 86M 34% /.tether
tmpfs 1.0M 0 1.0M 0% /.tether-init
rootfs 965M 0 965M 0% /lib/modules
/dev/disk/by-label/fe01ce2a7fbac8fa 976M 1.3M 908M 1% /demo
Let’s now create a file in the volume in question, and make sure the data is persistent:

root@3ef0b682bc8d:/# cd /demo
root@3ef0b682bc8d:/demo# echo "important" >> need-to-persist.txt
root@3ef0b682bc8d:/demo# cat need-to-persist.txt
root@3ef0b682bc8d:/demo# cd ..
root@3ef0b682bc8d:/# exit
Now launch a new container (a simple busybox image) with the same volume, and ensure that the data that we created is still accessible and persistent.

root@photon [ /workspace/vic ]#  docker -H --tls run \
-v demo:/demo -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
a3ed95caeb02: Pull complete
8ddc19f16526: Pull complete
Digest: sha256:65ce39ce3eb0997074a460adfb568d0b9f0f6a4392d97b6035630c9d7bf92402
Status: Downloaded newer image for library/busybox:latest

/ # df
Filesystem           1K-blocks      Used Available Use% Mounted on
devtmpfs                987544         0    987544   0% /dev
tmpfs                  1026584         0   1026584   0% /dev/shm
tmpfs                  1026584       132   1026452   0% /run
tmpfs                  1026584         0   1026584   0% /sys/fs/cgroup
/dev/sda               8125880     19612   7670456   0% /
tmpfs                   131072     43940     87132  34% /.tether
tmpfs                     1024         0      1024   0% /.tether-init
                        999320      1288    929220   0% /demo
/ # cd /demo
/demo # ls
lost+found           need-to-persist.txt
/demo # cat need-to-persist.txt
/demo #

There you have it – docker volumes in VIC. One final note is what happens when the VCH is deleted. If the VCH is deleted, then the docker volumes associated with that VCH are also deleted (although I also believe that we will change this behavior in future versions):

root@photon [ /workspace/vic ]# ./vic-machine-linux delete -t 
'administrator@vsphere.local:zzzzz@' \
--compute-resource Mgmt --name VCH01
INFO[2016-07-14T14:36:26Z] ### Removing VCH ####
INFO[2016-07-14T14:36:26Z] Removing VMs
INFO[2016-07-14T14:36:29Z] Removing images
INFO[2016-07-14T14:36:29Z] Removing volumes
INFO[2016-07-14T14:36:29Z] Removing appliance VM network devices
INFO[2016-07-14T14:36:30Z] Removing VCH vSphere extension
INFO[2016-07-14T14:36:35Z] Removing Resource Pool VCH01
INFO[2016-07-14T14:36:35Z] Completed successfully
root@photon-NaTv5i8IA [ /workspace/vic ]#

So for now, be careful if you place data in a volume, and then remove the VCH as this will also remove the volumes.

OK – one final test. Let’s assume that you did not use the “default” label, or that you had multiple volume store specified in the command line (which is perfectly acceptable). How then would you select the correct datastore for the volume? Let’s deploy a new VCH, and this time we will set the label to NFS:

root@photon [ /workspace/vic ]#  ./vic-machine-linux create \
--bridge-network Bridge-DPG --image-datastore isilion-nfs-01 \
-t 'administrator@vsphere.local:VMware123!@' \
 --compute-resource Mgmt --external-network VMNW51 --name VCH01 \
--volume-store "NFS:isilion-nfs-01/docker-vols"
INFO[2016-07-14T14:43:04Z] ### Installing VCH ####
INFO[2016-07-14T14:43:04Z] Generating certificate/key pair - private key in ./VCH01-key.pem
INFO[2016-07-14T14:43:05Z] Validating supplied configuration
INFO[2016-07-14T14:43:05Z] Firewall status: DISABLED on /CNA-DC/host/Mgmt/
INFO[2016-07-14T14:43:05Z] Firewall configuration OK on hosts:
INFO[2016-07-14T14:43:05Z]   /CNA-DC/host/Mgmt/
INFO[2016-07-14T14:43:05Z] License check OK on hosts:
INFO[2016-07-14T14:43:05Z]   /CNA-DC/host/Mgmt/
INFO[2016-07-14T14:43:05Z] DRS check OK on:
INFO[2016-07-14T14:43:05Z]   /CNA-DC/host/Mgmt/Resources
INFO[2016-07-14T14:43:06Z] Creating Resource Pool VCH01
INFO[2016-07-14T14:43:06Z] Datastore path is [isilion-nfs-01] docker-vols
INFO[2016-07-14T14:43:06Z] Creating appliance on target
INFO[2016-07-14T14:43:06Z] Network role management is sharing NIC with client
INFO[2016-07-14T14:43:06Z] Network role external is sharing NIC with client
INFO[2016-07-14T14:43:08Z] Uploading images for container
INFO[2016-07-14T14:43:08Z]      bootstrap.iso
INFO[2016-07-14T14:43:08Z]      appliance.iso
INFO[2016-07-14T14:43:13Z] Registering VCH as a vSphere extension
INFO[2016-07-14T14:43:18Z] Waiting for IP information
INFO[2016-07-14T14:43:44Z] Waiting for major appliance components to launch
INFO[2016-07-14T14:43:44Z] Initialization of appliance successful
INFO[2016-07-14T14:43:44Z] Log server:
INFO[2016-07-14T14:43:44Z] DOCKER_HOST=
INFO[2016-07-14T14:43:44Z] Connect to docker:
INFO[2016-07-14T14:43:44Z] docker -H --tls info
INFO[2016-07-14T14:43:44Z] Installer completed successfully

Now we want to create a volume, but place it in the folder/datastore identified by the label NFS. First, let’s try to create a volume as before:

root@photon [ /workspace/vic ]# docker -H --tls --tls \
volume create --name=demo --opt Capacity=1024
Error response from daemon: Server error from Portlayer: [POST /storage/volumes/][500]\

Yes – we know it is a horrible error message, and we will fix that. But to make it work, you now need another option to docker volume called VolumeStore. Here it is.

root@photon [ /workspace/vic ]# docker -H --tls \
 volume create --name=demo --opt Capacity=1024 --opt VolumeStore=NFS
root@photon [ /workspace/vic ]#

Now you can consume the volume in the same way as it was shown in the previous example.

Caution: A number of commands shown here will definitely change in future releases of VIC. However, what I have shown you is how to get started with docker volumes in VIC v0.4.0. If you do run into some anomalies that are not described in the post, and you feel it is a mismatch in behavior with standard docker, please let me know. I will feed this back to our engineering team, who are always open to suggestions on how to make VIC as seamless as possible to standard docker behavior.