{"id":5432,"date":"2017-11-21T08:31:13","date_gmt":"2017-11-21T16:31:13","guid":{"rendered":"\/developerblog\/?p=5432"},"modified":"2020-03-14T18:25:41","modified_gmt":"2020-03-15T01:25:41","slug":"autoscaling-deep-learning-training-kubernetes","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/autoscaling-deep-learning-training-kubernetes\/","title":{"rendered":"Autoscaling Deep Learning Training with Kubernetes"},"content":{"rendered":"

Background<\/h2>\n

We recently partnered with Litbit<\/a>, a San Jose-based startup, on a project to autoscale deep learning training. Litbit enables its customers to turn their “Internet of Things\u201d into conscious personas that can learn, think, and do helpful things. In order to accomplish this goal, customers train their AI-empowered personas using sight, sound, and touch sensors (among others) to recognize specific situations.<\/p>\n

Since different customers may be training different AI personas at different times, the training load tends to be bursty and unpredictable. Some of these training jobs (e.g., Spark ML) make heavy use of CPUs, while others (e.g., TensorFlow) make heavy use of GPUs. In the latter case, some jobs retrain a single layer of the neural net and finish very quickly, while others need to train an entire new neural net\u00a0and can take several hours to days.<\/p>\n

To meet the diverse requirements for training in a cost-efficient manner, Litbit needed a system that could scale different types of VM pools (CPU only, light GPU, heavy GPU) up and down based on demand. In this code story, we have generalized lessons learned from this scenario and will explain how to use the acs-engine-autoscaler<\/em> to scale different types of VMs up and down based on demand.<\/p>\n

Overview<\/h2>\n

While there are many options for running containerized distributed deep learning at scale, we have selected Kubernetes due to its superior cluster management technology and the huge developer community. To start, we need to create a Kubernetes cluster with GPU support on Azure to run different types of machine learning loads. Then we need to add autoscaling capability to the Kubernetes cluster to meet bursty demands in a cost-efficient manner.<\/p>\n

Creating a Kubernetes cluster with GPU support using ACS-engine<\/h2>\n

To create a Kubernetes cluster that supports GPUs, we will use acs-engine<\/em>, an open source tool that will generate the ARM template we need to deploy our cluster with everything already configured.<\/p>\n

NOTE: You might be wondering why we are using\u00a0acs-engine<\/em> and not\u00a0AKS<\/a> (Azure Container Service, the managed Kubernetes service on Azure). To use the acs-engine-autoscaler<\/em>, the Kubernetes cluster must be created by\u00a0acs-engine<\/em> as the autoscaler requires metadata information about the agent pools to scale the nodes up and down, which is information only exposed by\u00a0acs-engine.\u00a0<\/em>Therefore, the acs-engine-autoscaler<\/em> does not work with AKS.<\/p>\n

Install<\/h3>\n

Binary downloads for the latest version of acs-engine<\/em><\/a>\u00a0are available. Download acs-engine<\/em> for your operating system. Extract the binary and copy it to your $PATH<\/em>.<\/p>\n

Generate Templates<\/h3>\n

acs-engine<\/em> reads a JSON cluster definition which describes the size, shape, and configuration of your cluster.<\/p>\n

First, update example\\kubernetes.json<\/a> to create a cluster that satisfies our requirements. To create different types of VM pools (CPU only, light GPU, heavy GPU), we can create different agent pools by adding additional sections under agentPoolProfiles<\/em>. Each pool can have a different VM size and can scale up to 100 nodes (as set by the MaxAgentCount<\/em>\u00a0constant in acs-engine). For our scenario, we want a cluster for training with GPUs and inference with CPUs only. Here we are defining two pools as we don’t want to pay for GPU unless needed. The number of agents isn\u2019t really important because we are going to enable autoscaling later, so we will keep everything as 1.<\/p>\n

At the time of this writing, Azure has 6 different VM sizes with GPU support<\/a>.\u00a0For more details with generating templates, refer to the\u00a0ACS engine guide<\/a>.<\/p>\n

Here is an example of our Kubernetes cluster definition with multiple agent pools:<\/p>\n

{\r\n  \"apiVersion\": \"vlabs\",\r\n  \"properties\": {\r\n    \"orchestratorProfile\": {\r\n      \"orchestratorType\": \"Kubernetes\"\r\n    },\r\n    \"masterProfile\": {\r\n      \"count\": 1,\r\n      \"dnsPrefix\": \"mymlcluster\",\r\n      \"vmSize\": \"Standard_D2_v2\"\r\n    },\r\n    \"agentPoolProfiles\": [\r\n      {\r\n        \"name\": \"agentpool1\",\r\n        \"count\": 1,\r\n        \"vmSize\": \"Standard_D2_v2\",\r\n        \"availabilityProfile\": \"AvailabilitySet\"\r\n      },     \r\n      {\r\n        \"name\": \"agentpool2\",\r\n        \"count\": 1,\r\n        \"vmSize\": \"Standard_NC6\",\r\n        \"availabilityProfile\": \"AvailabilitySet\"\r\n      }\r\n    ],\r\n    \"linuxProfile\": {\r\n      \"adminUsername\": \"azureuser\",\r\n      \"ssh\": {\r\n        \"publicKeys\": [\r\n          {\r\n            \"keyData\": \"SSH-PUB-KEY\"\r\n          }\r\n        ]\r\n      }\r\n    },\r\n    \"servicePrincipalProfile\": {\r\n      \"clientId\": \"xxxxx\",\r\n      \"secret\": \"xxxxx\"\r\n    }\r\n  }\r\n}<\/pre>\n
Now with the cluster definition JSON, let’s generate the templates by running:<\/p>\n
$ acs-engine generate examples\/kubernetes.json<\/pre>\n
This step generates a bunch of files under the _output\/mymlcluster directory, including the ARM template and parameters that we want.<\/p>\n
Deploy Templates<\/h3>\n
With the new azuredeploy.json<\/em> and azuredeploy.parameters.json<\/em> generated in the previous step, we can now deploy the templates using the Azure CLI<\/a>.<\/p>\n
Note: <\/strong>make sure you choose a region that has N-series VM available<\/a>. For example, eastus and southcentralus are two regions with N-series skus available. Also, make sure your subscription has enough cores to run those VM types.<\/p>\n
$ cd _output\/mymlcluster\r\n$ az login\r\n$ az account set --subscription \"<SUBSCRIPTION NAME OR ID>\"\r\n$ az group create \\\r\n    --name \"<RESOURCE_GROUP_NAME>\" \\\r\n    --location \"<LOCATION>\"\r\n\r\n$ az group deployment create \\\r\n    --resource-group \"<RESOURCE_GROUP_NAME>\" \\\r\n    --template-file azuredeploy.json \\\r\n    --parameters azuredeploy.parameters.json<\/pre>\n
This step will take between 5 to 10 minutes to deploy. We will keep the generated azurdeploy.json<\/em> and azuredeploy.parameters.json<\/em> around as we will need them later to setup autoscaling.<\/p>\n
Once the deployment is completed, copy the Kubernetes config file of the cluster locally to allow kubectl to communicate with the cluster. If you do not already have the kubectl cli, follow these instructions<\/a> to install kubectl.<\/p>\n
$ scp azureuser@<dnsname>.<regionname>.cloudapp.azure.com:.kube\/config ~\/.kube\/config<\/pre>\n
\u00a0Verifying the Cluster<\/h3>\n
To ensure everything is working as intended, run:<\/p>\n
$ kubectl describe node <name-of-a-gpu-node><\/pre>\n
You should see the correct number of GPUs reported (in this example, it shows 1 GPU for a NC6 VM):<\/p>\n
...\r\nCapacity:\r\n alpha.kubernetes.io\/nvidia-gpu:    1\r\n cpu:                   6\r\n...<\/pre>\n
If alpha.kubernetes.io\/nvidia-gpu is shown as 0, wait a bit longer. The driver installation takes about 12 minutes, and the node might join the cluster before the installation is completed. After a few minutes, the node should restart, and report the correct number of GPUs.<\/p>\n
Scheduling a GPU Container<\/h3>\n
Now that we have a GPU-enabled Kubernetes cluster, we can run a container that requires GPU resources. Below is an example GPU container running TensorFlow. To request GPU resources, we have to specify how many GPU the container needs, then Kubernetes will map the device into the container. To use the drivers, we need to mount the driver from the Kubernetes agent host into the container.<\/p>\n
Note:<\/strong> the drivers are installed under \/usr\/lib\/nvidia-384\u00a0<\/em>(or another version number depending on the driver’s version).<\/p>\n
apiVersion: extensions\/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  labels:\r\n    app: tensorflow\r\n  name: tensorflow\r\nspec:\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: tensorflow\r\n    spec:     \r\n      containers:\r\n      - name: tensorflow\r\n        image: tensorflow\/tensorflow:latest-gpu\r\n        command: [\"python main.py\"]      \r\n        imagePullPolicy: IfNotPresent\r\n        env:\r\n        - name: LD_LIBRARY_PATH\r\n          value: \/usr\/lib\/nvidia:\/usr\/lib\/x86_64-linux-gnu\r\n        resources:\r\n          requests:\r\n            alpha.kubernetes.io\/nvidia-gpu: 1 \r\n        volumeMounts:\r\n        - mountPath: \/usr\/local\/nvidia\/bin\r\n          name: bin\r\n        - mountPath: \/usr\/lib\/nvidia\r\n          name: lib\r\n        - mountPath: \/usr\/lib\/x86_64-linux-gnu\/libcuda.so.1\r\n          name: libcuda\r\n      volumes:\r\n        - name: bin\r\n          hostPath: \r\n            path: \/usr\/lib\/nvidia-384\/bin\r\n        - name: lib\r\n          hostPath: \r\n            path: \/usr\/lib\/nvidia-384\r\n        - name: libcuda\r\n          hostPath:\r\n            path: \/usr\/lib\/x86_64-linux-gnu\/libcuda.so.1<\/pre>\n
Note:<\/strong> we have specified alpha.kubernetes.io\/nvidia-gpu: 1<\/em> for the resources requests, and mounted the drivers from the host into the container. We also modified the LD_LIBRARY_PATH<\/em> environment variable to let Python know where to find the driver’s libraries.<\/p>\n
Some libraries, such as libcuda.so<\/em>, are installed under \/usr\/lib\/x86_64-linux-gnu<\/em> on the host. Depending on your requirements, you might need to mount them separately as shown above.<\/p>\n
Schedule the deployment with the following command:<\/p>\n
$ kubectl create -f tftrain.yaml<\/pre>\n
Autoscaling Kubernetes Cluster to Meet Bursty Demands<\/h2>\n
Now that we have a Kubernetes cluster that can run CPU workloads and GPU workloads, we need to be able to scale the VM pools up and down based on demands. \u00a0Kubernetes-acs-engine-autoscaler<\/a>, a fork of OpenAI’s Kubernetes-ec2-autoscaler, can autoscale an acs-engine Kubernentes cluster based on demand.<\/p>\n
The Kubernetes<\/em>–acs-engine-autoscaler<\/em> will run inside the cluster and monitor the different pods that get scheduled. Whenever a pod is pending because of a lack of resources, the autoscaler will create an adequate number of new VMs to support the scheduled pod. Finally, when VMs are idle, the autoscaler will delete them. As a result, we can achieve the flexibility we want, while still keeping costs down.<\/p>\n
Setting up the Autoscaler<\/h3>\n
The\u00a0acs-engine-autoscaler<\/em> can be installed with a Helm chart. Helm is a Kubernetes package manager that helps us package, install, and manage our Kubernetes applications. Using the stable\/acs-engine-autoscaler<\/a> Helm chart, we can install the autoscaler in our cluster.<\/p>\n
First, locate your azuredeploy.parameters.json<\/em>\u00a0file generated with acs-engine<\/em>\u00a0from the previous step.<\/p>\n
Next, find the\u00a0values.yaml<\/a> file from the acs-engine-autoscaler Helm chart. Update the following parameters in the file.<\/p>\n
acsenginecluster:\r\n  resourcegroup:\r\n  azurespappid:\r\n  azurespsecret:\r\n  azuresptenantid:\r\n  kubeconfigprivatekey:\r\n  clientprivatekey:\r\n  caprivatekey:<\/pre>\n\n\n\n\n\n\n\n\n\n\n
Parameter<\/td>\nDescription<\/td>\n<\/tr>\n
resourcegroup<\/td>\nName of the resource group containing the cluster<\/td>\n<\/tr>\n
azurespappid<\/td>\nAn Azure service principal ID<\/td>\n<\/tr>\n
azurespsecret<\/td>\nAn Azure service principal secret<\/td>\n<\/tr>\n
azuresptenantid<\/td>\nAn Azure service principal tenant ID<\/td>\n<\/tr>\n
kubeconfigprivatekey<\/td>\nThe key passed to the kubeConfigPrivateKey<\/em> parameter in your azuredeploy.parameters.json<\/em> generated with acs-engine<\/em><\/td>\n<\/tr>\n
clientprivatekey<\/td>\nThe key passed to the clientPrivateKey<\/em>\u00a0parameter in your azuredeploy.parameters.json<\/em> generated with acs-engine<\/em><\/td>\n<\/tr>\n
caprivatekey<\/td>\n\u00a0The key passed to the caPrivateKey<\/em>\u00a0parameter in your azuredeploy.parameters.json<\/em> generated with acs-engine<\/em><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Finally, after you have updated the values.yaml file in the chart, run the following to install the chart with the release name my-release:<\/p>\n
$ helm install stable\/acs-engine-autoscaler<\/pre>\n
\u00a0Verifying Installation<\/h3>\n
To verify the acs-engine-autoscaler<\/em> is configured properly, find the pod that the deployment created and look at its logs. The result will look something similar to the following:<\/p>\n
To verify that acs-engine-autoscaler has started, run:\r\n\r\n  kubectl --namespace=default get pods -l \"app=olfactory-bunny-acs-engine-autoscaler\"\r\n\r\nTo verify that acs-engine-autoscaler is running as expected, run:\r\n  kubectl logs $(kubectl --namespace=default get pods -l \"app=olfactory-bunny-acs-engine-autoscaler\" -o jsonpath=\"{.items[0].metadata.name}\")\r\n\r\n$ kubectl --namespace=default get pods -l \"app=olfactory-bunny-acs-engine-autoscaler\"\r\n\r\nNAME                                                     READY     STATUS    RESTARTS   AGE\r\nolfactory-bunny-acs-engine-autoscaler-1715934483-c673v   1\/1       Running   0          10s\r\n\r\n$ kubectl logs $(kubectl --namespace=default get pods -l \"app=olfactory-bunny-acs-engine-autoscaler\" -o jsonpath=\"{.items[0].metadata.name}\")<\/pre>\n
You should see something like the following in the logs of the autoscaler pod.<\/p>\n
2017-06-11 23:20:59,352 - autoscaler.cluster - DEBUG - Using kube service account\r\n2017-06-11 23:20:59,352 - autoscaler.cluster - INFO - ++++ Running Scaling Loop ++++++\r\n2017-06-11 23:20:59,421 - autoscaler.cluster - INFO - Pods to schedule: 0\r\n2017-06-11 23:20:59,421 - autoscaler.cluster - INFO - ++++ Scaling Up Begins ++++++\r\n2017-06-11 23:20:59,421 - autoscaler.cluster - INFO - Nodes: 1\r\n2017-06-11 23:20:59,421 - autoscaler.cluster - INFO - To schedule: 0\r\n2017-06-11 23:20:59,421 - autoscaler.cluster - INFO - Pending pods: 0\r\n2017-06-11 23:20:59,422 - autoscaler.cluster - INFO - ++++ Scaling Up Ends ++++++\r\n2017-06-11 23:20:59,422 - autoscaler.cluster - INFO - ++++ Maintenance Begins ++++++\r\n2017-06-11 23:20:59,422 - autoscaler.engine_scaler - INFO - ++++ Maintaining Nodes ++++++\r\n2017-06-11 23:20:59,423 - autoscaler.engine_scaler - INFO - node: k8s-agentpool1-29744472-4                                                   state: under-utilized-undrainable\r\n2017-06-11 23:20:59,423 - autoscaler.cluster - INFO - ++++ Maintenance Ends ++++++\r\n...<\/pre>\n
Autoscaling the Cluster<\/h2>\n
Recall from the previous section how our Kubernetes cluster has 2 agent pools, each with a single agent. The job we ran in the previous section requires 1 GPU and only agentpool2 has a VM with GPU. Now to test our autoscaler, let’s schedule a second job with GPU request that is similar to the tftrain.yaml deployment we ran earlier in our cluster.<\/p>\n
From the autoscaler’s pod’s log, we should see agentpool2 scaling up to meet the demand:<\/p>\n
autoscaler.cluster - INFO - ++++++ Running Scaling Loop ++++++\r\nautoscaler.cluster - INFO - Pods to schedule: 1\r\nautoscaler.cluster - INFO - ++++++ Scaling Up Begins ++++++\r\nautoscaler.cluster - INFO - Nodes: 2\r\nautoscaler.cluster - INFO - To schedule: 1\r\nautoscaler.cluster - INFO - Pending pods: 1\r\nautoscaler.cluster - INFO - ========= Scaling for 1 pods ========\r\n[...]\r\nautoscaler.cluster - INFO - New capacity requested for pool agentpool2: 2 agents (current capacity: 1 agents)\r\nautoscaler.deployments - INFO - Deployment started<\/pre>\n
After a few minutes, the new VM with GPU will be created, and our second job starts running. Once the jobs are completed, the pods are terminated.\u00a0The autoscaler will notice one or more nodes are now idle and will adjust the cluster size accordingly.<\/p>\n
First, idle VMs will be cordoned and drained:<\/p>\n
autoscaler.cluster - INFO - node: k8s-agentpool1-32238962-1                                                   \r\nstate: under-utilized-drainable\r\nautoscaler.kube - INFO - cordoned k8s-agentpool1-32238962-1\r\nautoscaler.kube - INFO - Deleting Pod kube-system\/kube-proxy-ghr3z\r\nautoscaler.kube - INFO - drained k8s-agentpool1-32238962-4<\/pre>\n
Then after some time, the cordoned node will get deleted:<\/p>\n
autoscaler.cluster - INFO - node: k8s-agentpool1-32238962-1                                                   \r\nstate: idle-unschedulable\r\nautoscaler.container_service - INFO - deleting node k8s-agentpool1-32238962-1\r\nautoscaler.container_service - INFO - Deleting VM\r\nautoscaler.container_service - INFO - Deleting NIC\r\nautoscaler.container_service - INFO - Deleting OS disk<\/pre>\n
Voil\u00e0! Now we have a Kubernetes cluster that can autoscale as new pods are scheduled and resources are requested.<\/p>\n
Horizontal Pod Autoscaling<\/h2>\n
In some scenarios, you might want to\u00a0scale up and down based on some metrics, for example, CPU or memory usage.\u00a0Kubernetes\u00a0Horizontal Pod Autoscaling<\/a>\u00a0(HPA)\u00a0allows us to specify a metric and target to track on a deployment.<\/p>\n
For example, for a given\u00a0deployment, you might want to configure HPA to have a combined average CPU usage not exceeding 50%. Once the CPU usage of all running pods exceeds 50%, HPA will increase the number of replicas in the deployment and spread the load across the cluster. But eventually, the existing VMs in the cluster will not be able to support more replicas, and new pods created by HPA will start hanging in a <pending> state. This is where the acs-engine-autoscaler<\/em> will notice the pending pods and start to create new VMs to support them, then delete the idle VMs once the jobs are completed.<\/p>\n
To understand how to configure Horizontal Pod Autoscaling,\u00a0check out the official documentation<\/a>.<\/p>\n
Conclusion<\/h2>\n
With this solution, we were able to help Litbit to scale up to 40 nodes at a time then subsequently downscale as planned. Litbit has been successfully using this for the past 4 months.\u00a0 This solution is ideal for use cases where you need to scale different types of VMs up and down based on demand. To test the Azure autoscaler for your own use case, check out this GitHub repo<\/a>.<\/p>\n
Resources<\/h2>\n