Photon Controller – Image Issues – 413 Request Entity Too Large

PHOTON_square140Over the last couple of days I’ve been getting to grips with Photon Controller v0.8. For those of you who do not follow developments in our Cloud Native Apps BU, Photon Controller leverages ESXi hosts to provide compute and management for containers at large scale. It will also allow you to stand up container frameworks such as Kubernetes, Docker Swarm and Mesos very quickly. I’m not going to take you through the step-by-step instructions on how to do this as my colleague and good pal William Lam has already done this. Instead, I’m going to try to highlight some newbie issues that I have run into. Expect a few more of these over the coming weeks.

*** Please note that at the time of writing, Photon Controller is still not GA ***

*** The steps highlighted here may change in the GA version of the product ***

The first significant roadblock I hit was trying to upload my first image (an OVA) using Photon CLI on my Windows desktop. I would use this image for future VM deployments. On attempting to create this new image, I set the target to the photon controller and I ran the following command and hit the following issue.

C:\>photon -n image create Z:\isos\VMware\Photon\1.0TP2\photon-1.0TP2.ova \
-n photon-1.0TP2.ova -i EAGER
2016/05/04 12:07:25 Error: photon: HTTP 413: <html>
<head><title>413 Request Entity Too Large</title></head>
<body bgcolor="white">
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.9.3</center>
</body>
</html>
C:>

The clues here are “413 Request Entity Too Large” and the reference to nginx (pronounced “EngineX”). Nginx is a web server. Now the troubleshooting began. If you want to find out the root cause, you can jump to the end of the post. But I think there are some interesting items in the post for those of you just getting started with Photon Controller. After some googling, it would seem that this is a result of a limitation with the Nginx configuration. My first step was to figure out which container on the Photon Controller was running Nginx. To do that, I needed to see the list of containers. This can be seen from the following photon CLI command (output truncated for readability):

esxcloud [ ~ ]$ docker ps -a
CONTAINER ID        IMAGE                     COMMAND                  CREATED             
cfe46ee5ce08        esxcloud/installer_ui     "/bin/sh -c 'sh /etc/"   22 hours ago ..       
d260061ab81f        esxcloud/cloud-store      "bash /etc/esxcloud/r"   22 hours ago ..      
d8b9680754e1        esxcloud/housekeeper      "bash /etc/esxcloud/r"   22 hours ago ..      
8357de068897        esxcloud/deployer         "bash /etc/esxcloud/r"   22 hours ago ..     
b47775e07d39        esxcloud/management-api   "bash /etc/esxcloud/r"   22 hours ago ..      
39ca85833448        esxcloud/zookeeper        "bash /etc/esxcloud/r"   22 hours ago ..      
As you see, there are a number of different containers running on the photon controller. So which one is responsible for the Xnginx web server? The next command lets me find this quite quickly:
esxcloud [ ~ ]$ docker inspect cfe46ee5ce08 | grep nginx
        "sh /etc/nginx/config.sh \u0026\u0026 nginx -g 'daemon off;'"
            "Destination": "/etc/nginx/certs",
            "Destination": "/var/cache/nginx",
            "sh /etc/nginx/config.sh \u0026\u0026 nginx -g 'daemon off;'"
            "/etc/nginx/certs": {},
            "/var/cache/nginx": {}
esxcloud [ ~ ]$ 

It would seem that Nginx web server is running in the install_ui container. OK – good. Now lets attach to that container with a bash shell, and take a look around. You can do this as follows:

esxcloud [ ~ ]$ docker exec -i -t cfe46ee5ce08 /bin/bash
root@esxcloud-management-vm:/#

Now that I had a bash shell in the container running Nginx, I was able to look around. The first step was to check out the nginx.conf configuration file, and see if the parameter mentioned by a few folks as the root cause was present and set. This parameter is called client_max_body_size, and if it is set too low, it can cause the symptoms that I saw. But I got to wondering .. why had William not seen this issue. We had both used the same OVA for our image. Anyway, I didn’t think it would hurt to add this parameter to the configuration file and restart Nginx to see if it fixed the issue. However, there was no editor in this container so I had to get that installed too. This involved running the following commands (I’ve truncated the output from the commands):

root@esxcloud-management-vm:/# apt-get update
.
.
root@esxcloud-management-vm:/# apt-get install vim
.
.

Now that I had the “vi” editor, I went ahead and added the entry to the config file and restarted Nginx. However I still could not upload an image. I then decided to compare all of the steps that I carried out against those of William’s guide, and then I spotted something. William has set the target for his Photon CLI to be [IP address]:28080. I had simply set mine to [IP address]. The 28080 port is the port of the load balancer. Should it make any difference? I don’t know, so I decided to set my target to include the load balancer port. Success! I could now create images.

What was the difference? Well, I eventually found an answer in the issues section of the photon controller on github. It looks like I was not the only one to encounter this. Michael West provided an explanation which I will paraphrase here:

The image create request that I was originally making defaulted to port 80. This is the management UI which as we just saw is proxied by Nginx. Nginx is now acting as the HTTP proxy for the Photon Controller API. If Nginx doesn’t like the HTTP request body size then Nginx kills the request, not the Photon Controller API. The Photon Controller API runs on port 9000, or if you want to go through the load balancer, port 28080. In that case you will not see Nginx in the path.

I hope this helps you if you run into the same situation.

On a side note, I’ll be taking some time with the Cloud Native Apps team over the coming months. This is part of a Take-3 experience provided by VMware, where employees get to spend 3 months doing something different from their normal day-to-day. Expect similar posts to this as I get up to speed.

One comment

Comments are closed.