Background
Webjet is an online travel agency responsible for managing thousands of travel bookings daily in Australia and New Zealand. Over the last year, Microsoft and Webjet have collaborated to help Webjet solve inefficiencies in the processing and management of hotel bookings by leveraging blockchain technology. You can learn more about our partnership from the press release. For more technical details watch this talk from Build 2017.
As Webjet continued to grow their blockchain deployment, they began to hit several issues with their Infrastructure-as-a-Service (IaaS) architecture. We recently worked with Webjet to help migrate their deployment to Kubernetes. In this code story, we’ll share the lessons learned from migrating from IaaS to Kubernetes, explain how we used Helm to deploy a private Ethereum network to Kubernetes, and show how you can use Helm to deploy your own private Ethereum network.
The Road to Kubernetes
Due to Virtual Machine Scale Sets’ (VMSS) feature of autoscaling based on a variety of metrics, Webjet chose to host their private Ethereum networks on a series of VMSS with each one running Docker containers orchestrated by Docker Compose. A generalized version of Webjet’s Azure ARM deployment is available on GitHub. However, in Webjet’s path to moving this to production, some deficiencies in their IaaS-based architecture became apparent.
Difficult to scale
Docker Compose supports running multiple replicas for a container; however, in order to load balance across replicas, management of container-host port mappings is required.
At the node-level, as VMSS’ autoscale feature scales out the entire node, VMSS work optimally against homogeneous workloads. With several different services running on the machine, scaling up/down a VMSS would affect all running services on that machine. Alternatively, the architecture could be separated into several Docker Compose files and thereby several VMSS, but this would require a significant amount of pre-planning to ensure a proper balance of hardware needs and costs.
Error-prone upgrades
Upgrading services meant SSH-ing into the VMs, running docker-compose stop, downloading the new Docker Compose manifest, and then subsequently running docker-compose up. Configuration management tools such as SaltStack could be used to automate this process, but the process itself is very custom and has a high potential for error.
For the above reasons, the Webjet team decided to leverage Kubernetes through AKS to help automate deployment, scaling, and orchestration of the application. With Kubernetes, Webjet is able to scale up/down certain applications and take advantage of Kubernetes’ native support for rolling upgrades.
Architecture
After migrating their existing Docker Compose manifests to Kubernetes, we landed on the following architecture:
The diagram above shows the architecture of the private Ethereum network when mapped to Kubernetes constructs. The blue represents Kubernetes Services, the red represents Kubernetes Deployments, whereas the green and yellow are Kubernetes Secrets and ConfigMaps respectively.
Every time Webjet needed to deploy a new Ethereum network, a series of configuration updates needed to be made to the YAML definitions. These changes included the Ethereum genesis file, application secrets, and Geth private keys. Webjet initially accomplished this task using shell scripts that would search and replace configuration values and then execute the deployment using kubectl create. During a hackfest with Webjet, we worked together to instead use Helm to manage the templatization of the YAML definitions.
Helm
Helm is a package manager for Kubernetes. The project was initially created by Deis and has since been donated to the Cloud Native Computing Foundation (CNCF). Sidenote, the lovely folks of Deis are now part of Microsoft. Typically, an application is composed of a combination of services, deployments, secrets, etc. Instead of managing these Kubernetes resources individually, Helm offers a higher-level construct (known as charts) to manage your entire application. With Helm, you can create, upgrade, and rollback entire applications, and easily share applications/charts with peers and the greater community.
Creating a Chart
Working with Webjet, we created a Helm chart to deploy a private Ethereum network. To create a Helm chart, install the Helm CLI, and run helm create to scaffold a new chart. To migrate your existing YAML manifests, copy the manifests to the Helm folder structure and add the following labels outlined here: Chart Standard Labels. These labels are not required to deploy charts but are recommended for consistency. The next step is to templatize configurations and settings that can be overridden. For more details, see the Chart Best Practices Guide.
With Helm, each chart is an individually-manageable unit. Webjet separated their architecture into multiple Helm charts such that they could be individually upgradeable. For instance, we had separate charts for the Ethereum network (Miners, EthStats, Bootnode), another for the Blockchain Watcher, one for deploying their Storage (SQL), etc. Artifacts that were shared across multiple Helm charts (e.g., Secrets, ConfigMaps) were marked as dependencies in the Chart’s requirement.yaml.
Deploying a Chart
Helm is comprised of two components: a client CLI (helm) and a server (tiller). One of the niceties of deploying a Kubernetes cluster on Azure through either acs-engine or ACS/AKS, is that by default the cluster will be pre-provisioned with Tiller. For a cluster that has not yet been initialized, you can do so with helm init.
To install a chart, you can run helm install, which will search for and install a chart from the official Charts repository. The Ethereum chart we developed in concert with Webjet is available on GitHub, and once this pull request is merged, you’ll be able to install a private Ethereum network onto your Kubernetes cluster using:
helm install incubator/ethereum
Until the pull request is merged, I have made available the Chart archive:
helm install https://japoonhelmstrg.blob.core.windows.net/public/ethereum-0.1.0.tgz --set geth.account.publicKey=[PUBLIC_KEY] --set geth.account.privateKey=[PRIVATE_KEY] --set geth.account.secret=[SECRET]
Note that geth.account.publicKey, geth.account.privateKey, and geth.account.secret are required configurations. To create a new Geth account, please refer to https://github.com/ethereum/go-ethereum/wiki/Managing-your-accounts.
You now have a private Ethereum network running inside your Kubernetes cluster.
To clarify, this chart deploys a private Ethereum network that is not connected to MainNet.
With Webjet, their charts were committed to a Git repository and all deployments were initiated with Helm by supplying a local filepath to the Chart. However, a private Helm Charts repository can be easily provisioned to host your private Charts.
Summary
This code story explains how Webjet migrated their deployment to Kubernetes and shows how Helm can be applied to simplify the deployment and management of applications hosted on Kubernetes. Using Kubernetes on Azure has greatly simplified Webjet’s deployment process. Our solution using Helm removed the need for Webjet to manage their own deployment scripts and makes it possible to leverage the many existing charts made available by the community.
Charts can be submitted to the Charts registry and shared with the community. The Ethereum chart shown in the article is open-sourced, and there is an open pull request to add it to the Charts registry. This Chart deploys a simplified private Ethereum network; in order to deploy a network in a more highly available manner, please refer to this chart. If you have any questions or feedback on this code story, please reach out in the comments below.
0 comments