{"id":6667,"date":"2018-08-09T07:58:25","date_gmt":"2018-08-09T14:58:25","guid":{"rendered":"\/developerblog\/?p=6667"},"modified":"2020-03-20T07:25:26","modified_gmt":"2020-03-20T14:25:26","slug":"infrastructure-code-demand-gpu-clusters-terraform-jenkins","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/infrastructure-code-demand-gpu-clusters-terraform-jenkins\/","title":{"rendered":"Infrastructure as Code – On demand GPU clusters with Terraform & Jenkins"},"content":{"rendered":"

Background<\/b><\/h2>\n

Developing robust algorithms for self-driving cars requires sourcing event data from over 10 billion hours of recorded driving time. But even with 10 billion-plus hours, it can be challenging to capture rare yet critical edge cases like weather events or collisions at scale. Luckily, these rare events can often be simulated effectively.<\/p>\n

Cognata<\/a>, a startup that develops an autonomous car simulator uses patented computer vision and deep learning algorithms to automatically generate a city-wide simulation that includes buildings, roads, lane marks, traffic signs, and even trees and bushes.<\/p>\n

Simulating driving event data, however, requires expensive GPU and compute resources that vary depending on a given event.\u00a0In working with\u00a0Cognata<\/a>, the big obstacle was going to be automation and creating on-demand resources in Azure.<\/p>\n

In this code story, we’ll describe how we were able to provision custom GPU rendering clusters on demand with a Jenkins pipeline and Terraform.<\/p>\n

The Challenges<\/b><\/h2>\n

Cognata required a scalable architecture to render its simulations across individual customers and events using GPU. To support this need, we first investigated using Docker containers and Kubernetes. However, at the time of this post, nvidia-docker does not officially support X server<\/a> and OpenGL<\/a> support is a beta feature.\u00a0 As an alternative to container orchestration, we architected a scalable solution using a GPU\u00a0Azure Virtual Machine Scale Sets<\/a>\u00a0(VMSS) cluster and a custom application installation script for on-demand deployment.<\/p>\n

The complex changesets must be applied to the infrastructure with minimal human interaction and must support\u00a0building, changing, and versioning infrastructure safely and efficiently.<\/p>\n

The solution<\/b><\/h2>\n

On-demand Infrastructure as Code<\/h2>\n

Cognata’s applications and services are deployed in a Kubernetes cluster and the rendering application is deployed in GPU Virtual machines clusters.\u00a0 To apply “Infrastructure as code” methodology, we decided to use Terraform<\/a> and Jenkins<\/a>.<\/p>\n

Terraform allows us to provision, deprovision, and orchestrate immutable infrastructure in a declarative manner; meanwhile, Jenkins pipelines offers delivery process rather than an \u201copinionated\u201d\u00a0process and allows us to analyze and optimize the process.<\/p>\n

Jenkins offered two big advantages:<\/p>\n

    \n
  1. The pipelines can be triggered via the HTTP API on demand.<\/li>\n
  2. Jenkins in Kubernetes is very easy. We used the Jenkins Helm chart:\n
    $ helm install --name jenkins stable\/jenkins<\/pre>\n<\/li>\n<\/ol>\n
    Deploying and Destroying resources<\/h2>\n
    Creating on-demand resource required us to create detailed plans and specification of our resources. These plans are stored in a git repository such as GitHub or BitBucket to maintain versioning and continuity.<\/p>\n
    In our case, the plans included a Virtual Machines Scale Sets (VMSS) with GPU and an installation script of Cognata’s application.\u00a0 The VMSS was created from a predefined image that included all the drivers and prerequisites of the application.<\/p>\n
    Once we have the plans<\/a> in the git repository, we need to run the workflow in the Jenkins pipeline to deploy the resources.<\/p>\n
    Starting the workflow can be triggered from an HTTP request to the Jenkins pipeline.<\/p>\n
    http:\/\/JENKINS_SERVER_ADDRESS\/job\/YOUR_JOB_NAME\/buildWithParam<\/pre>\n
    After the pipeline was triggered, the plans were pulled from the git repository and the deploying process is started using Terraform.<\/p>\n
    Terraform works in 3 stages:<\/p>\n
      \n
    1. Init –\u00a0The\u00a0terraform init<\/code>\u00a0command is used to initialize a working directory containing Terraform configuration files.<\/li>\n
    2. Plan –\u00a0The\u00a0terraform plan<\/code>\u00a0command is used to create an execution plan.<\/li>\n
    3. Apply –\u00a0The\u00a0terraform apply<\/code>\u00a0command is used to apply the changes required to reach the desired state of the configuration, or the pre-determined set of actions generated by a\u00a0terraform plan<\/code>\u00a0execution plan.<\/li>\n<\/ol>\n
      Once the plan was executed and Terraform saved the state results in files, w<\/span>e needed to upload the result files to a persistent storage such as Azure Blob storage<\/a>\u00a0. This step would later enable us to download the state files and destroy the resources that we created after the business process was completed, and to actually\u00a0create on-demand clusters.<\/span><\/p>\n
      The flow of the solution is described in the following image:<\/p>\n
      <\/p>\n
      Terraform client<\/h3>\n
      To invoke Terraform commands in the Jenkins pipeline, we created a small Docker container with Terraform and Azure CLI with the following Dockerfile.<\/p>\n
      FROM azuresdk\/azure-cli-python:hotfix-2.0.41\r\n\r\nARG tf_version=\"0.11.7\"\r\n\r\nRUN apk update && apk upgrade && apk add ca-certificates && update-ca-certificates && \r\n    apk add --no-cache --update curl unzip\r\n\r\nRUN curl https:\/\/releases.hashicorp.com\/terraform\/${tf_version}\/terraform_${tf_version}_linux_amd64.zip -o terraform_${tf_version}_linux_amd64.zip && \r\n    unzip terraform_${tf_version}_linux_amd64.zip -d \/usr\/local\/bin && \r\n    mkdir -p \/opt\/workspace && \r\n    rm \/var\/cache\/apk\/*\r\n\r\nWORKDIR \/opt\/workspace\r\nENV TF_IN_AUTOMATION somevalue<\/pre>\n
      Conclusion and Reuse<\/b><\/h2>\n
      With the adoption of driverless cars, 10 billion-plus hours of recorded driving time just isn’t viable. Cognata’s complex simulations rendering for multiple autonomous car manufacturers can become costly and inefficient. Our Jenkins pipeline and Terraform solution enabled Cognata to dynamically scale GPU resources for their simulations, making it easier to serve their customers while saving significant cost in compute resources. Using these technologies we were able to automate the deployment and maintenance of Cognata’s Azure VMSS GPU clusters and simulation logic.<\/p>\n
      Our joint solution is adaptable to any workload that requires:<\/p>\n