Mesos on Photon Controller

mesosAnother framework that can be very quickly stood up on Photon Controller is Mesos. Apache Mesos is yet another cluster framework for container orchestration and availability (yes, there are many). The steps for deploying the Photon Controller Installer, deploying Photon Controller and creating the tenants, resource tickets and projects are identical to those outlines in steps 1,2 and 3 of my Docker SWARM on Photon Controller post. There is no point in repeating all of the steps here. I will highlight some of the other steps involved in deploying Mesos on Photon Controller, but I don’t really want to focus on those as they are a repeat of what is in the SWARM article, though slightly different. Instead, I want to show you what you can do after Mesos is deployed – a sort of “what do I do now that I have Mesos deployed?” approach. William Lam also showed you how to get started with Mesos and Photon Controller here, but did not have the resources to show Mesos in action. Since I have the resources to show you how to run apps/containers via Mesos, that’s what I will demonstrate in this post.

*** Please note that at the time of writing, Photon Controller is still not GA ***

*** The steps highlighted here may change in the GA version of the product ***

To begin, lets highlight some of the  differences between deploying Docker Swarm versus Apache Mesos on Photon Controller.

  1. Create a MESOS image

We obviously need a Mesos image for this deployment. We did this with Swarm previously. This is how to create one for Mesos (same syntax, but different image file – which can be located here).

C:\Users\chogan>photon image create Downloads\photon-mesos-vm-disk1.vmdk \
-n mesos-vm.vmdk -i EAGER
Using target 'http://10.27.44.34:28080'
Created image 'mesos-vm.vmdk' ID: 07da5493-6904-47ef-acc8-6a1edbb0b6a1

C:\Users\chogan>

As highlighted in the Swarm post, some arguments need an identifier (-n, name and -i, type) but the path to the image does not. Also, monitor the upload of the image to the image datastore and make sure it completes before proceeding to the next step. The -i EAGER option means that it will be uploaded immediately, rather than the ON_DEMAND option, which means it is uploaded when it is first required.

2. Allow deployment to support MESOS cluster

C:\Users\chogan>photon deployment enable-cluster-type \
7c2941f4-5f93-495d-843a-693c6106111e -k mesos \
-i 07da5493-6904-47ef-acc8-6a1edbb0b6a1
Are you sure [y/n]? y
Using target 'http://10.27.44.34:28080'
Cluster Type: MESOS
Image ID: 07da5493-6904-47ef-acc8-6a1edbb0b6a1
C:\Users\chogan>

The first id is that of the deployment. The second is  that of the MESOS image uploaded previously.

3. Create a Mesos Cluster

In this example I have requested that 3 slaves get created.

C:\Users\chogan>photon cluster create -n mesos -k mesos --dns 10.16.142.110 \
--gateway 10.27.47.254 --netmask 255.255.240.0 --zookeeper1 10.27.44.35 -s 3
Using target 'http://10.27.44.34:28080'
Zookeeper server 2 static IP address (leave blank for none):

Creating cluster: mesos (MESOS)
  Slave count: 3

Are you sure [y/n]? y
C:\Users\chogan>

4. Verify the state of the Mesos Cluster

There are a few commands that can be run to check the state of the cluster. The second one below provides the IP address of the Mesos master(s) as well as the IP address of the Marathon node. We’ll talk more about the Marathon node shortly.

C:\Users\chogan>photon cluster list
Using target 'http://10.27.44.34:28080'
ID                                    Name   Type   State  Slave Count
ee99b4c3-7e7b-44ab-ab0a-0c5c569cd2e2  mesos  MESOS  READY  3

Total: 1
READY: 1

This command will display a listing of all the nodes:

C:\Users\chogan>photon cluster list_vms ee99b4c3-7e7b-44ab-ab0a-0c5c569cd2e2
Using target 'http://10.27.44.34:28080'
ID                                    Name                                            \
State
39c163bc-d9e2-446f-a051-8af8bca6c903  master-51f68b39-cd0a-433b-a65d-c57e79ae0313     \
STARTED
4c32c456-a91e-47c3-903a-f811ca470c31  slave-6223e09a-7626-433e-ab60-0b8288dc0c3e      \
STARTED
5608c7b0-00f7-41bf-8bf1-617f407426d9  zookeeper-45f96ded-5095-47f4-8ae0-734ae09d0e15  \
STARTED
69eb5f0f-44b2-43ef-8626-1afd69e82a95  marathon-d58d9917-770c-480f-8642-3715568f51eb   \
STARTED
9dc4a999-2bf1-4699-97f0-45ba52436493  master-3aeb5ef7-d696-4bd1-80d7-c6456594d160     \
STARTED
d5289e47-4d5e-405a-8bd4-22774210879c  slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4      \
STARTED
d659cd73-85da-4b17-b032-4b834656e708  master-76ab4165-8c22-4f61-b196-62dddea06ab5     \
STARTED
f329b59f-cad8-41cc-a32e-d786e418f53c  slave-b2198eb8-75cb-40f6-a814-e837f4f19000      \
STARTED

Total: 8
STARTED: 8

This next command will provide the IP addresses if the constituent components.

C:\Users\chogan>photon cluster show ee99b4c3-7e7b-44ab-ab0a-0c5c569cd2e2
Using target 'http://10.27.44.34:28080'
Cluster ID:             ee99b4c3-7e7b-44ab-ab0a-0c5c569cd2e2
  Name:                 mesos
  State:                READY
  Type:                 MESOS
  Slave count:          3
  Extended Properties:  map[gateway:10.27.47.254 netmask:255.255.240.0 \
  dns:10.16.142.110 zookeeper_ips:10.27.44.35]

VM ID                                 VM Name                                         \
VM IP
39c163bc-d9e2-446f-a051-8af8bca6c903  master-51f68b39-cd0a-433b-a65d-c57e79ae0313     \
10.27.33.211
5608c7b0-00f7-41bf-8bf1-617f407426d9  zookeeper-45f96ded-5095-47f4-8ae0-734ae09d0e15  \
10.27.44.35
69eb5f0f-44b2-43ef-8626-1afd69e82a95  marathon-d58d9917-770c-480f-8642-3715568f51eb   \
10.27.33.219
9dc4a999-2bf1-4699-97f0-45ba52436493  master-3aeb5ef7-d696-4bd1-80d7-c6456594d160     \
10.27.33.212
d659cd73-85da-4b17-b032-4b834656e708  master-76ab4165-8c22-4f61-b196-62dddea06ab5     \
10.27.33.209

C:\Users\chogan>

You will notice that there are 3 masters. This is for Mesos high availability. One master is elected the leader. If it fails, another takes over.

The zookeeper node is responsible for coordinating the election of the master leader, and informing the other nodes about who the leader is.

Management Containers in Mesos

You can query the containers running in Mesos by logging into each node, and running a “docker ps -a” command.

Here is the marathon node:

root@marathon-d58d9917-770c-480f-8642-3715568f51eb [ ~ ]# docker ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED     \
        STATUS              PORTS               NAMES
9eb90142e94f        mesosphere/marathon:v0.15.0   "./bin/start --master"   18 hours ago\
        Up 18 hours                             hopeful_yalow
root@marathon-d58d9917-770c-480f-8642-3715568f51eb [ ~ ]#

Here is a master node:

root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160 [ ~ ]# docker ps -a
CONTAINER ID        IMAGE                                               \
COMMAND                  CREATED             STATUS          PORTS     NAMES
08827e917164        mesosphere/mesos-master:0.26.0-0.2.145.ubuntu1404   \
"mesos-master --regis"   19 hours ago        Up 19 hours              photon-mesos-master
root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160 [ /etc ]#

Here is a slave node:

root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4 [ ~ ]# docker ps -a
CONTAINER ID        IMAGE                                              COMMAND        \
     CREATED             STATUS              PORTS               NAMES
35ace556e3cc        mesosphere/mesos-slave:0.26.0-0.2.145.ubuntu1404   "mesos-slave"  \
     18 hours ago        Up 18 hours                             photon-mesos-slave
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4 [ ~ ]#

Mesos is not like docker swarm, insofar as that there are no docker commands to give you an overview of the whole cluster. Instead, you can log into each of the nodes, query the running containers, and attach to them by running a bash commands if you wish. By typing in mesos and tab completing, you can get a list of mesos specific commands on the master:

root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160 [ ~ ]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
08827e917164 mesosphere/mesos-master:0.26.0-0.2.145.ubuntu1404 \
"mesos-master --regis" 19 hours ago Up 19 hours photon-mesos-master
root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160 [ ~ ]# docker exec -i \
-t 08827e917164 bash

root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160:/# mesos<tab complete>
mesos mesos-init-wrapper mesos-ps mesos-start-cluster.sh mesos-stop-masters.sh
mesos-cat mesos-local mesos-resolve mesos-start-masters.sh mesos-stop-slaves.sh
mesos-daemon.sh mesos-log mesos-scp mesos-start-slaves.sh mesos-tail
mesos-execute mesos-master mesos-slave mesos-stop-cluster.sh

root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160:/# exit
exit
root@master-3aeb5ef7-d696-4bd1-80d7-c6456594d160 [ ~ ]#

Getting started with Marathon

There are now two ways that we can manage and monitor the Mesos cluster. The first is via the Marathon interface (http://<marathon_ip>:8080). The second is via the leading Master interface (http://<master_ip>:5050). Note that you can connect to any of the master nodes, and you will be automatically directed to the leader.

This is what you should see when you point a web browser at the Marathon node:

marathon_uiClick on the button to create an application. We will create a simple command that will sleep, and then write a message to a temporary file. To do so, give the application a name, some CPU and Memory resources, leave instances at the default of 1, and in thecommand section, type in the following:

sleep 5; echo "Mesos running on Photon Controller - w00t!" >> /tmp/output.txt

Finally, click on the “Create” button at the bottom:

my-marathon-appYou should now see the app running:

marathon-app-runningAnd if we switch over to the master view (port 5050), we can also see the task regularly running:

mesos master viewAnd to close this example, we can log onto one of the slaves nodes, connect to the slave  container, and examine the “/tmp/output” file that we are updating via the command.

root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4 [ ~ ]# docker ps
CONTAINER ID        IMAGE                                              COMMAND         \
    CREATED             STATUS              PORTS               NAMES
35ace556e3cc        mesosphere/mesos-slave:0.26.0-0.2.145.ubuntu1404   "mesos-slave"   \
    19 hours ago        Up 19 hours                             photon-mesos-slave
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4 [ ~ ]# docker exec -i -t 35ace556e3cc \
/bin/bash
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4:/# more /tmp/output.txt
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
Mesos running on Photon Controller - w00t!
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4:/#
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4:/# exit
exit
root@slave-6b0e4744-4792-4812-b0ea-f48d4cdd66f4 [ ~ ]#

Of course, this is only scratching the surface of what one can do with Mesos. However the idea once again is to show how the underlying resources of a number of ESXi hosts can very quickly and easily be consumed  using Photon Controller, and how quickly a Mesos cluster can be stood up.