May 31st, 2017

A Production Deis Workflow Deployment on Azure

James Spring
James Spring

Background

Deis Workflow is the second version of the Deis application platform that allows for the easy deployment and scalability of Twelve-Factor applications. Previously, we worked with Deis to enable their V1 PaaS on Azure based on CoreOS.

With Deis Workflow, the only requirement for installation is supporting Kubernetes. In November of 2016, the Azure Container Service announced support for Kubernetes on Azure. With this addition, we arranged another hackfest with the team from Deis.

The primary goal of the hackfest was to run the Deis Workflow End-to-End tests successfully. We accomplished this goal within the first few hours of the three-day event, which allowed time to explore other topics, including one that a customer was interested in: what a “production Deis Workflow deployment on Azure” would look like and the steps necessary to get there.

Deis Workflow, in addition to providing a platform for Twelve-Factor applications, simplifies working with Kubernetes.  This allows developers to more readily focus on their apps initially without having to learn how to operate Kubernetes proper.  As needed, or as familiarity improves, operating directly with Kubernetes is possible.  Workflow and Kubernetes can be used without interference between the two.

The customer was interested in taking their current use of Kubernetes and Workflow for internal development and testing, and see what would be needed to move that setup into production.  The definition of “production” in this context of Azure will rely on Deis’ documentation of a production deployment. The key points are:

  • Internal state is moved to Azure Blob Store
  • Update registration mode
  • Enable TLS using a Platform certificate

Below we will cover the steps necessary get Deis Workflow up and running on Azure. We will also address the production deployment steps mentioned above in depth.

Requirements

To set up, configure, and install Deis Workflow, we will need to install a set of tools:

  • Azure CLI: allows for interaction with various Azure services from the CLI.
  • kubectl: primary tool for interacting with a Kubernetes cluster.
  • Deis Workflow CLI: tool for interacting with Deis Workflow.
  • Helm: Kubernetes package manager.

During the hackfest, the Azure Container Service had a certificate issue (since resolved) that required we use one additional tool:

  • acs-engine: the open source engine for creating Azure Resource Manager (ARM) templates for deploying clusters (Docker Swarm, Kubernetes, and DC/OS) onto Azure

To fulfill the recommendations within the Deis Production Deployments document, we will need to:

  • Provision an Azure Storage account to move Workflow’s internal state into persistent storage
  • Procure a wildcard certificate to enable TLS

Setup

Below, we outline the needed steps and pointers to getting the tooling set up.

Third Party Tools

Each of the following third party tools, as mentioned above, need to be installed:

To verify each is properly configured:

jims@tupperware:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
jims@tupperware:~$ deis --version
v2.9.1
jims@tupperware:~$ helm version
Client: &version.Version{SemVer:"v2.1.2", GitCommit:"58e545f47002e36ca71ac5d1f7a987b56e1937b3", GitTreeState:"clean"}
Error: cannot connect to Tiller

Azure CLI

The Azure CLI will be used to setup the necessary infrastructure components within Azure.  The different options for installing the Azure CLI are here.  Once installed, we will need to log in, which is a two-step process.  First, you issue the az login command and then visit the Device Login page and enter the token presented on the command line.

jims@tupperware:~$ az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code BXSABJN4S to authenticate.
[
  {
    "cloudName": "AzureCloud",
    "id": "ab12ec88-8e28-41ed-8537-5e17766001f5",
    "isDefault": true,
    "name": "Developers Research",
    "state": "Enabled",
    "tenantId": "cfd488bf-86f1-41af-91ab-2d7cd011db47",
    "user": {
      "name": "jaspring@microsoft.com",
      "type": "user"
    }
  }
]

When you go to Device Login, you will be prompted to enter the code shown in the above step. You may also need to log into your Azure account, as well.

At this point, you are ready to use the Azure CLI.

acs-engine

As mentioned previously, during the hackfest the ACS resource provider generated a certificate that did not play well with Deis Workflow.  The certificate contained a value in the Subject Alternative Name field that the HTTP library Workflow used did not like.  So, acs-engine was used to provision the cluster.

The steps necessary to install acs-engine are here.

Deis Workflow can be installed on any Kubernetes cluster, so one can use the Azure Portal or the Azure CLI Azure Container Service commands to provision a Kubernetes cluster on Azure in place of using acs-engine.

Preparation

With the tooling in place, there is some additional preparation needed, which is detailed in the following sections. You’ll need to:

  • Create a Resource Group
  • Generate an SSH Key
  • Create a Service Principal
  • Determine the details of the Kubernetes cluster
  • Create a storage account and retrieve its key for the preservation of Deis Workflow’s state
  • Create a certificate to enable TLS on Deis Workflow.

Create a Resource Group

A Resource Group is necessary for all Azure Resource Manager resources. It requires a name and a region to which it is deployed. For this tutorial, we will use:

  • Name: deisprodk8s
  • Location: westus
jims@tupperware:~$ az group create --location="westus" --name="deisprodk8s"
{
  "id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s",
  "location": "westus",
  "managedBy": null,
  "name": "deisprodk8s",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null
}

Generate an SSH Key

An SSH Key is needed to access machines in the Kubernetes cluster. The generation of keys will vary from platform to platform (for example, for Ubuntu, one can look here). For this tutorial, we will create a 2048 bit RSA key and store it in the file ~/.ssh/id_k8s_rsa:

jims@tupperware:~$ ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_k8s_rsa -C jaspring@microsoft.com
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jims/.ssh/id_k8s_rsa.
Your public key has been saved in /home/jims/.ssh/id_k8s_rsa.pub.
The key fingerprint is:
SHA256:HW7ptbn212QsD/VTWcZRNVtrl9ckFjmzkYVbqSe9WG8 jaspring@microsoft.com
The key's randomart image is:
+---[RSA 2048]----+
|              +X@|
|             .B+%|
|          .   o&*|
|         o o o+==|
|        S = . =o=|
|         o . +ooE|
|          . o  Bo|
|            ..  +|
|           ..... |
+----[SHA256]-----+

For the Kubernetes configuration, the public key will be needed:

jims@tupperware:~$ more ~/.ssh/id_k8s_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqM
F7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDd
wf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjm
KqvvuL3gZaQ4QCHz7 jaspring@microsoft.com

Create a Service Principal

A Service Principal is an entity that is granted permissions to perform operations within an Azure subscription. Kubernetes uses a Service Principal to manipulate compute resources as needed/defined during interactions with the Kubernetes cluster.

To create a Service Principal, use the Azure CLI:

jims@tupperware:~$ az ad sp create-for-rbac
{
  "appId": "ad349d5e-2b0e-49ce-beaa-407c68196774",
  "name": "http://azure-cli-2016-12-23-16-35-16",
  "password": "b77abcf0-2f83-4e83-bc1a-c2c7dac8d9be",
  "tenant": "123488bf-86f1-41af-92cb-2d7cd011db47"
}

Kubernetes Cluster details

For the cluster, we need additional information, like:

  • Resource group: this is the grouping under which all resources will be mapped in Azure
  • Region: which Azure Data Center to create the cluster in
  • DNS Prefix: a prefix used to name the public components of the Kubernetes cluster
  • Agent Machine Count: how many agent machines are needed
  • Agent Machine Type: what type of VMs to use for the Kubernetes cluster
  • Administrator Name: the name of the administrative user account on cluster nodes

For this deployment, we will use the following values:

  • Resource group: deisprodk8s
  • Region: westus (West US)
  • DNS Prefix: deisprod
  • Agent Machine Count: 4
  • Agent Machine Type: Standard_D2_v2 (a general compute-oriented VM with 7GB RAM and 100GB local disk)
  • Administrator Name: dadmin

acs-engine uses a configuration file to generate the templates necessary for deployment. Using the above information, including the Service Principal and the SSH Key, the configuration file ~/flow/k8s/config/kubernetes.json resembles the following:

bash
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "deisprod",
      "vmSize": "Standard_D2_v2"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 4,
        "vmSize": "Standard_D2_v2",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "dadmin",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "servicePrincipalClientID": "ad349d5e-2b0e-49ce-beaa-407c68196774",
      "servicePrincipalClientSecret": "b77abcf0-2f83-4e83-bc1a-c2c7dac8d9be"
    }
  }
}

Create Storage Account and Retrieve Key

The storage account will be created in the same Resource Group as defined above, deisprodk8s. The settings we will use for the storage account are:

  • Storage Account Name: deisprodconfig
  • Storage Account SKU: Standard_LRS (Standard, Locally Redundant Storage)
  • Storage Account Kind: Storage

Creating the storage account:

jims@tupperware:~$ az storage account create --sku="Standard_LRS" --resource-group="deisprodk8s" --kind="Storage" --location="westus" --name="deisprodconfig"
{
  "accessTier": null,
  "creationTime": "2016-12-23T17:10:32.334380+00:00",
  "customDomain": null,
  "encryption": null,
  "id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Storage/storageAccounts/deisprodconfig",
  "kind": "Storage",
  "lastGeoFailoverTime": null,
  "location": "westus",
  "name": "deisprodconfig",
  "primaryEndpoints": {
    "blob": "https://deisprodconfig.blob.core.windows.net/",
    "file": "https://deisprodconfig.file.core.windows.net/",
    "queue": "https://deisprodconfig.queue.core.windows.net/",
    "table": "https://deisprodconfig.table.core.windows.net/"
  },
  "primaryLocation": "westus",
  "provisioningState": "Succeeded",
  "resourceGroup": "deisprodk8s",
  "secondaryEndpoints": null,
  "secondaryLocation": null,
  "sku": {
    "name": "Standard_LRS",
    "tier": "Standard"
  },
  "statusOfPrimary": "Available",
  "statusOfSecondary": null,
  "tags": {},
  "type": "Microsoft.Storage/storageAccounts"
}

Retrieving the storage account keys:

jims@tupperware:~$ az storage account keys list --name="deisprodconfig" --resource-group="deisprodk8s"
{
  "keys": [
    {
      "keyName": "key1",
      "permissions": "FULL",
      "value": "f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ=="
    },
    {
      "keyName": "key2",
      "permissions": "FULL",
      "value": "ad9pMFJUualKDH3irkNyS3XXH0xMbs0JC56ESwiVnRaCJUnuc/6FEQf09xzjsK4vO34LNq+4VV1N20DI9kau5g=="
    }
  ]
}

Create a Certificate to Enable TLS

There are two approaches you can take to enable TLS on Deis Workflow. The first is to create an application certificate; for instance, if all of the public traffic will come through www.example.com, then only a certificate specific to www.example.com would be needed. These certificates are typically referred to as domain-specific certificates. Within Deis, these might be referred to as application-specific certificates.

A more general approach is to use what is referred to as a wildcard certificate. The domains specified in the wildcard certificate would resemble *.example.com. Thus any name prefixing .example.com would match. This certificate can be installed in Deis Workflow as a platform certificate.

For development and testing, you can use OpenSSL to generate a self-signed certificate. Digital Ocean has a good tutorial.  Another option is to consider is Let’s Encrypt.

For production, certificates typically cost money. Their retrieval can be quick or take a while, depending on the type of certificate used.

In this tutorial, we will request a PositiveSSL Wildcard certificate from Namecheap.

To get the certificate issued, you must create a Certificate Signing Request (CSR). Certificates have several fields that can be specified for the CSR. In this case, we will specify the following:

  • Country (C): US
  • State (ST): California
  • Locality (L): Santa Cruz
  • Organization (O): plusonetechnology.net
  • Common Name (CN): *.plusonetechnology.net

The Common Name field is what a TLS connection is verified against when a connection comes into the server.

To create the CSR:

jims@tupperware:~$ openssl req -new -newkey rsa:2048 -nodes 
> -out star_plusonetechnology_net.csr 
> -keyout star_plusonetechnology_net.key 
> -subj "/C=US/ST=California/L=Santa Cruz/O=plusonetechnology.net/CN=*.plusonetechnology.net"
Generating a 2048 bit RSA private key
................................+++
.....+++
writing new private key to 'star_plusonetechnology_net.key'
-----

The CSR is written as a PEM blob; in this case, it is written to the file star_plusonetechnology_net.csr (contents shown below):

jims@tupperware:~$ more star_plusonetechnology_net.csr
-----BEGIN CERTIFICATE REQUEST-----
MIICvjCCAaYCAQAweTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx
EzARBgNVBAcTClNhbnRhIENydXoxHjAcBgNVBAoTFXBsdXNvbmV0ZWNobm9sb2d5
Lm5ldDEgMB4GA1UEAxQXKi5wbHVzb25ldGVjaG5vbG9neS5uZXQwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDoudQXqW5++L5a1fUtBNNlXuLeUOJII+Ci
10MdWWUosyn1hHku0AcBXHdONDLfq2W1jUMrxlXaDWKSDaM1xLUCVVKfiK9tpM/v
erhVhddUATHE8ZSXPY/HZGkCCoJlF88FDM+aKieteQn6mN7EjemvhtNB2suy455S
WKosuo/oHRmGuI+CfqIf7hVq1TFzGd2Bd4NWmkldyJp2EvbN+2gr+5G8yXNExwiv
answPtLZD9oKs4ZmWZP17r7YizDgHdhSONZdtqOt0WhBeJOB+44hIwEzKtMzzZwZ
R29MiQaBy42wSXq71AUONfOKdTBmGUf9QQLMexFiew4+UFjiwi8pAgMBAAGgADAN
BgkqhkiG9w0BAQUFAAOCAQEAcyuepBBHSHNh4zlCIhHcnQnTni1COHMx/7DIaeTq
UCJm4Yt26n77EwdK/b4l97l09PIBOdZowf9ETSgLvd2jy7KmbMFQMuRZmH1jZwSa
Cj8DV8PixA4qm9ARBAugQmwoEFVPmxWCC+lD7O8ti+bolP6twS/dsjM3bPCu1l23
5X8pqHfrYPI/C28INKdr2TdUFQi0ocFIqm3RkYhfGjOrTii9m99enf3HFFX9Pr8z
p1QjSlewyOrkiXcQIAP1Fnck48txFCyU5tFsyTBoF6y91jR9GY5wL/kDwvxbFiWq
dBY3knjwtaKe/iLPL7YV2NNUA/+dJ014NJW3oeJRnnfQlQ==
-----END CERTIFICATE REQUEST-----

For issuing the certificate, Namecheap will require that the CSR be pasted into the website. It will also require some personal information and a process to validate that the domain specified in the certificate is owned by the person submitting the request. Since this process will depend on where the certificate is purchased and the CA used, the sequence will be skipped.

For the CSR for plusonetechnology.net, via Namecheap, DNS-based validation was used.

Provisioning the Kubernetes Cluster

Previously, we defined and updated the acs-engine configuration file as ~/flow/k8s/config/kubernetes.json. When acs-engine is run against this file, it will generate the necessary Azure Resource Manager (ARM) templates for deploying the Kubernetes cluster:

jims@tupperware:~/flow/k8s/config$ acs-engine -artifacts ~/flow/k8s/output ~/flow/k8s/config/kubernetes.json
cert creation took 7.712984265s
wrote /home/jims/flow/k8s/output/apimodel.json
wrote /home/jims/flow/k8s/output/azuredeploy.json
wrote /home/jims/flow/k8s/output/azuredeploy.parameters.json
...
wrote /home/jims/flow/k8s/output/kubeconfig/kubeconfig.westus.json
...
acsengine took 7.743813684s

Note: the ellipses (…) imply content that has been cut to keep the output shown more concise.

The next step is to deploy the templates. Again, the Resource Group we are using is deisprodk8s.

jims@tupperware:~/flow/k8s/config$ az group deployment create --resource-group="deisprodk8s" --template-file=/home/jims/flow/k8s/output/azuredeploy.json --parameters @/home/jims/flow/k8s/output/azuredeploy.parameters.json
{
  "id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Resources/deployments/azuredeploy",
  "name": "azuredeploy",
  "properties": {
    "correlationId": "a58352c4-b97d-4e89-bf23-e10e0c9637da",
    "debugSetting": null,
    "dependencies": [
      ...
    ],
    "mode": "Incremental",
    "outputs": {
      "masterFQDN": {
        "type": "String",
        "value": "deisprod.westus.cloudapp.azure.com"
      }
    },
    "parameters": {
      ...
    },
    "parametersLink": null,
    "providers": [
      ...
    ],
    "provisioningState": "Succeeded",
    "template": null,
    "templateLink": null,
    "timestamp": "2016-12-23T21:54:04.178783+00:00"
  },
  "resourceGroup": "deisprodk8s"
}

Setting Up `kubectl`

With the Kubernetes cluster deployed into Azure, we can now configure settings so that kubectl can talk to the cluster and we can interact with things.

When acs-engine was run, a number of kubeconfig.*.json files were generated for each region. Since our example deployment is in westus, the one we will look at is kubeconfig.westus.json. The KUBECONFIG environment variable needs to be set and its status checked to utilize this file, as shown below:

jims@tupperware:~/flow/k8s/output$ export KUBECONFIG=/home/jims/flow/k8s/output/kubeconfig/kubeconfig.westus.json
jims@tupperware:~/flow/k8s/output$ kubectl cluster-info
Kubernetes master is running at https://deisprod.westus.cloudapp.azure.com
Heapster is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use kubectl cluster-info dump.

At this point, kubectl is configured and we can interact with the cluster. Please note that adding the export KUBECONFIG to one’s startup files will be helpful for subsequent logins to the machine from which one would interact with the cluster.  You can also copy the configuration file to ${HOME}/.kube/config.

Kubernetes Using the `az` Command line

A much simpler way to install the Kubernetes cluster from the command line is available. For completeness, the method to deploy the same Kubernetes cluster is documented below.  The assumption is that the configuration is the same as what we specified in the kubernetes.json file used by acs-engine. That is:

  • Resource group: deisprodk8s
  • Region: westus
  • DNS Prefix: deisprod
  • Agent Machine Count: 4
  • Agent Machine Type: Standard_D2_v2
  • Administrator Name: dadmin
  • SSH Key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com
jims@tupperware:~$ az acs create --name="deisprod" 
> --orchestrator-type="Kubernetes" 
> --resource-group="deisprodk8s" 
> --admin-username="dadmin" 
> --agent-count=4 
> --agent-vm-size="Standard_D2_v2" 
> --dns-prefix="deisprod" 
> --location="westus" 
> --ssh-key-value="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com"
creating service principal.........done
waiting for AAD role to propogate.done
{
  "id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Resources/deployments/azurecli1482540765.0587363",
  "name": "azurecli1482540765.0587363",
  "properties": {
    "correlationId": "b1a52fec-8b35-49e7-af8a-e0d7ad9d056e",
    "debugSetting": null,
    "dependencies": [],
    "mode": "Incremental",
    "outputs": null,
    "parameters": {
      "clientSecret": {
        "type": "SecureString"
      }
    },
    "parametersLink": null,
    "providers": [
      {
        "id": null,
        "namespace": "Microsoft.ContainerService",
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiVersions": null,
            "locations": [
              "westus"
            ],
            "properties": null,
            "resourceType": "containerServices"
          }
        ]
      }
    ],
    "provisioningState": "Succeeded",
    "template": null,
    "templateLink": null,
    "timestamp": "2016-12-24T00:59:18.590702+00:00"
  },
  "resourceGroup": "deisprodk8s"
}

Installing Deis Workflow

Previously, both the Deis CLI and Helm were installed. With the KUBECONFIG variable set in the prior section, the steps needed to install Deis Workflow are straightforward. The only potential issue is that we need to set up the configuration to reflect the production concerns previously mentioned, which include:

  • Internal state/storage will use Azure Storage
  • Registration for users should be admin_only

Installing Deis requires the following initial steps:

  • Check your setup and initialize Helm 
    jims@tupperware:~/$ helm version
Client: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"}
Error: cannot connect to Tiller
jims@tupperware:~/$ helm init
$HELM_HOME has been configured at /home/jims/flow/helm.

Tiller (the helm server side component) has been installed into your Kubernetes Cluster.
Happy Helming!
jims@tupperware:~/$ helm version
Client: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"}
  • Add the Deis Workflow repository 
    jims@tupperware:~/$ helm repo add deis https://charts.deis.com/workflow
"deis" has been added to your repositories

During the install of Workflow, in order to meet the requirements laid out for a production system, some configuration information is needed.  Specifically, the information from the storage account we setup previously.  In addition, we need to know what config options are needed to lock down user registration to the admin_only.

Note: Since the hackfest, Deis Workflow defaults to registration being admin_only.

To configure and install Workflow, we need:

  • Storage Account Name: deisprodconfig
  • Storage Account Key: f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ==

And install Workflow as follows:

$ export STORAGE_ACCOUNT_NAME=deisprodconfig
$ export STORAGE_ACCOUNT_KEY="f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ==" 
$ helm install deis/workflow --namespace=deis 
> --set global.storage=azure,azure.accountname=$STORAGE_ACCOUNT_NAME,azure.accountkey=$STORAGE_ACCOUNT_KEY,azure.registry_container=registry,azure.database_container=database,azure.builder_container=builder
NAME:   whopping-ostrich
LAST DEPLOYED: Wed May 10 12:16:06 2017
NAMESPACE: deis
STATUS: DEPLOYED

RESOURCES:
==> extensions/v1beta1/Deployment
NAME                   DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
deis-monitor-grafana   1        1        1           0          3s
deis-logger-redis      1        1        1           0          3s
deis-database          1        1        1           0          3s
deis-controller        1        1        1           0          3s
deis-registry          1        1        1           0          3s
deis-monitor-influxdb  1        1        1           0          3s
deis-nsqd              1        1        1           0          3s
deis-logger            1        1        1           0          3s
deis-router            1        1        1           0          3s
deis-workflow-manager  1        1        1           0          3s
deis-builder           1        1        1           0          3s

==> v1/Secret
NAME                   TYPE    DATA  AGE
deis-router-dhparam    Opaque  1     4s
minio-user             Opaque  2     4s
objectstorage-keyfile  Opaque  5     4s

==> v1/ConfigMap
NAME                  DATA  AGE
slugrunner-config     1     4s
slugbuilder-config    2     4s
dockerbuilder-config  2     4s

==> v1/ServiceAccount
NAME                   SECRETS  AGE
deis-logger            1        4s
deis-router            1        4s
deis-builder           1        4s
deis-controller        1        4s
deis-registry          1        4s
deis-monitor-telegraf  1        4s
deis-nsqd              1        4s
deis-workflow-manager  1        4s
deis-logger-fluentd    1        4s
deis-database          1        4s

==> v1/Service
NAME                    CLUSTER-IP    EXTERNAL-IP  PORT(S)                                                   AGE
deis-controller         10.0.83.212          80/TCP                                                    4s
deis-registry           10.0.93.44           80/TCP                                                    4s
deis-nsqd               10.0.6.2             4151/TCP,4150/TCP                                         4s
deis-router             10.0.128.55       80:30854/TCP,443:30567/TCP,2222:31387/TCP,9090:32093/TCP  4s
deis-logger             10.0.55.43           80/TCP                                                    4s
deis-builder            10.0.206.67          2222/TCP                                                  4s
deis-workflow-manager   10.0.182.215         80/TCP                                                    4s
deis-monitor-grafana    10.0.39.43           80/TCP                                                    3s
deis-monitor-influxui   10.0.236.49          80/TCP                                                    3s
deis-database           10.0.184.196         5432/TCP                                                  3s
deis-monitor-influxapi  10.0.214.212         80/TCP                                                    3s
deis-logger-redis       10.0.146.66          6379/TCP                                                  3s

==> extensions/v1beta1/DaemonSet
NAME                   DESIRED  CURRENT  READY  NODE-SELECTOR  AGE
deis-monitor-telegraf  5        5        0               3s
deis-logger-fluentd    5        5        0               3s
deis-registry-proxy    5        5        0               3s

To verify the Deis Workflow deployment is complete, you can use the command kubectl –namespace=deis get pods. When installation is complete, output should resemble:

jims@tupperware:~$ kubectl --namespace=deis get pods
NAME                                     READY     STATUS    RESTARTS   AGE
deis-builder-587185222-rktvr             1/1       Running   0          6m
deis-controller-1274199582-3lqgv         1/1       Running   0          6m
deis-database-1755707874-9z55k           1/1       Running   0          6m
deis-logger-2533678197-91h6m             1/1       Running   0          6m
deis-logger-fluentd-69dqb                1/1       Running   4          6m
deis-logger-fluentd-bw2v7                1/1       Running   0          6m
deis-logger-fluentd-j5zt9                1/1       Running   0          6m
deis-logger-fluentd-mlddn                1/1       Running   0          6m
deis-logger-fluentd-sh8kw                1/1       Running   0          6m
deis-logger-redis-1307646428-w32k3       1/1       Running   0          6m
deis-monitor-grafana-59098797-2v6p2      1/1       Running   0          6m
deis-monitor-influxdb-168332144-5l7bx    1/1       Running   0          6m
deis-monitor-telegraf-624n4              1/1       Running   0          6m
deis-monitor-telegraf-6bmd6              1/1       Running   0          6m
deis-monitor-telegraf-bk833              1/1       Running   1          6m
deis-monitor-telegraf-x98sw              1/1       Running   0          6m
deis-monitor-telegraf-zzjjl              1/1       Running   0          6m
deis-nsqd-1042535208-b6th3               1/1       Running   0          6m
deis-registry-3958274866-qjv9h           1/1       Running   0          6m
deis-registry-proxy-58tv0                1/1       Running   0          6m
deis-registry-proxy-86rlw                1/1       Running   0          6m
deis-registry-proxy-9h3cz                1/1       Running   0          6m
deis-registry-proxy-djrts                1/1       Running   0          6m
deis-registry-proxy-mtc7d                1/1       Running   0          6m
deis-router-3258454730-5l05p             1/1       Running   0          6m
deis-workflow-manager-3582051402-1tzl6   1/1       Running   0          6m

Finalizing Deis Workflow Readiness

At this point, Deis Workflow is up and running on Kubernetes on Azure. However, we still have a few steps left to finalize the readiness of the deployment, including:

  • Setting up DNS for Deis Workflow
  • Registering the Deis Workflow Administrator
  • Installing the Deis Platform Certificate and Enable TLS

Setting up DNS for Deis Workflow

When applications are deployed to Workflow, a name can be specified, or one will be generated. That name is prefixed to the domain set up for Deis Workflow. Previously, a wildcard certificate was registered for the domain plusonetechnology.net. Azure DNS needs to be configured to point to Deis Workflow. We must do the following steps, detailed in the following sections:

  • Configure Azure DNS Zone for the domain name
  • Update DNS Entries for the domain name at the domain registrar
  • Test to verify DNS has updated

Configure Azure DNS

To configure Azure DNS you must:

  • Create the Azure DNS Zone
  • Create the Wildcard A Record-set
  • Create the Wildcard A Record

To create the zone:

jims@tupperware:~$ az network dns zone create --name="plusonetechnology.net" --resource-group="deisprodk8s"
{
  "etag": "00000002-0000-0000-3639-3f9f965dd201",
  "id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Network/dnszones/plusonetechnology.net",
  "location": "global",
  "maxNumberOfRecordSets": 5000,
  "name": "plusonetechnology.net",
  "nameServers": [
    "ns1-07.azure-dns.com.",
    "ns2-07.azure-dns.net.",
    "ns3-07.azure-dns.org.",
    "ns4-07.azure-dns.info."
  ],
  "numberOfRecordSets": 2,
  "resourceGroup": "deisprodk8s",
  "tags": {},
  "type": "Microsoft.Network/dnszones"
}

In the information returned, note the list of name servers. This list will be needed when updating the information for the domain at the domain registrar. This process will be covered in the next section.

Creating the record set:

jims@tupperware:~$ az network dns record-set a create 
> --name="*" 
> --zone-name="plusonetechnology.net" 
> --resource-group="deisprodk8s"
{
  "etag": "16fb3cb9-2a63-4ca2-babb-74e7a08fc9a6",
  "id": "/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/jmsdeis/providers/Microsoft.Network/dnszones/plusonetechnology.net/A/*",
  "metadata": null,
  "name": "*",
  "resourceGroup": "deisprodk8s",
  "ttl": 3600,
  "type": "Microsoft.Network/dnszones/A"
}

As part of the Deis Workflow setup, Deis provisions a load balancer for inbound traffic, referred to as the LoadBalancer Ingress, which is a public IP address. To set up the A-Record for the domain, we need to retrieve this address. Querying the Kubernetes cluster as shown below returns this information:

```bash
jims@tupperware:~$ kubectl --namespace=deis describe svc deis-router
Name:           deis-router
Namespace:      deis
Labels:         heritage=deis
Annotations:        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout=1200
Selector:       app=deis-router
Type:           LoadBalancer
IP:         10.0.128.55
LoadBalancer Ingress:   52.173.130.253
Port:           http    80/TCP
NodePort:       http    30854/TCP
Endpoints:      10.244.0.7:8080
Port:           https   443/TCP
NodePort:       https   30567/TCP
Endpoints:      10.244.0.7:6443
Port:           builder 2222/TCP
NodePort:       builder 31387/TCP
Endpoints:      10.244.0.7:2222
Port:           healthz 9090/TCP
NodePort:       healthz 32093/TCP
Endpoints:      10.244.0.7:9090
Session Affinity:   None
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  15m       15m     1   service-controller          Normal      CreatingLoadBalancer    Creating load balancer
  12m       12m     1   service-controller          Normal      CreatedLoadBalancer Created load balancer

As you can see above, the LoadBalancer Ingress IP address is 52.173.130.253. This value is used to set up the A-Record for Azure DNS:

jims@tupperware:~$ az network dns record-set a add-record --record-set-name="*" --resource-group="deisprodk8s" --zone-name="plusonetechnology.net" --ipv4-address=52.173.130.253
{
  "arecords": [
    {
      "ipv4Address": "52.173.130.253"
    }
  ],
  "etag": "99420e4d-8bfc-4b68-8509-1bd73f02c2ac",
  "id": "/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/jmsdeis/providers/Microsoft.Network/dnszones/plusonetechnology.net/A/*",
  "metadata": null,
  "name": "*",
  "resourceGroup": "jmsdeis",
  "ttl": 3600,
  "type": "Microsoft.Network/dnszones/A"
}

At this point, DNS is configured on the Azure side. The next step is to update the DNS information with the domain registrar.

Update DNS Entries for the Domain Name at the Domain Registrar

The method for updating DNS information will vary from registrar to registrar. The domain plusonetechnology.net is registered with Joker. Updating the DNS records for plusonetechnology.net at Joker will look like:

Note that the values in the image are the same nameserver values returned when we created the DNS Zone with the Azure CLI above.

Test to Verify DNS Has Updated

After changing that information on the domain registrar, we need to verify that the DNS has been updated. This process can take just a few minutes or it can be longer.

Since a Wildcard record was set up for the domain plusonetechnology.net, any name placed in from of the domain will now resolve to the same address.

jims@tupperware:~$ nslookup muppets.plusonetechnology.net
Server:        10.211.55.1
Address:    10.211.55.1#53

Non-authoritative answer:
Name:    muppets.plusonetechnology.net
Address: 52.173.130.25

At this point, DNS is properly configured, and the information has propagated.

Register the Deis Workflow Administrator

With a new Deis Workflow install, the first user registered is the administrator. To register:

jims@tupperware:~$ deis register http://deis.plusonetechnology.net
username: jims
password:
password (confirm):
email: jaspring@microsoft.com
Registered jims
Logged in as jims
Configuration file written to /home/jims/.deis/client.json
jims@tupperware:~$ deis users
=== Users (*=admin)
*jims

Install the Deis Platform Certificate and Enable TLS

Previously, a Wildcard Certificate was ordered from Namecheap. After going through the validation process, the certificates were delivered:

jims@tupperware:__plusonetechnology_net$ ls -ltra
total 40
drwx------@  5 jims  staff   170 Dec 23 13:24 .
drwx------+ 15 jims  staff   510 Dec 23 15:31 ..
-rwxr-xr-x@  1 jims  staff  7342 Dec 23 16:24 __plusonetechnology_net.p7b
-rwxr-xr-x@  1 jims  staff  1944 Dec 23 16:24 __plusonetechnology_net.crt
-rwxr-xr-x@  1 jims  staff  5630 Dec 23 16:24 __plusonetechnology_net.ca-bundle

This step is in addition to the private key and certificate signing request generated earlier.

Deis Workflow specifies how to enable Platform SSL. In short, the steps are:

  • Create a file tls.crt that has the TLS certificate, first, and chain certificates following
  • Create a file tls.key which is the private key file copied/renamed
  • Create a YAML file to update Deis Router settings
    • Secret name is deis-router-platform-cert
    • Requires both tls.crt and tls.key be base64 encoded
  • Use kubectl to add the new configuration

For tls.crt:

jims@tupperware:~/flow/deis/certs$ cat __plusonetechnology_net.crt __plusonetechnology_net.ca-bundle > tls.crt
jims@tupperware:~/flow/deis/certs$ more tls.crt
-----BEGIN CERTIFICATE-----
MIIFbTCCBFWgAwIBAgIRAJJIdJMChgmODD2ZZ/HKQQwwDQYJKoZIhvcNAQELBQAw
gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD
VQQDEy1DT01PRE8gUlNBIERvbWFpbiBWYWxpZGF0aW9uIFNlY3VyZSBTZXJ2ZXIg
Q0EwHhcNMTYxMjIzMDAwMDAwWhcNMTcxMjIzMjM1OTU5WjBkMSEwHwYDVQQLExhE
b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxHTAbBgNVBAsTFFBvc2l0aXZlU1NMIFdp
bGRjYXJkMSAwHgYDVQQDDBcqLnBsdXNvbmV0ZWNobm9sb2d5Lm5ldDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAOi51Bepbn74vlrV9S0E02Ve4t5Q4kgj
4KLXQx1ZZSizKfWEeS7QBwFcd040Mt+rZbWNQyvGVdoNYpINozXEtQJVUp+Ir22k
z+96uFWF11QBMcTxlJc9j8dkaQIKgmUXzwUMz5oqJ615CfqY3sSN6a+G00Hay7Lj
nlJYqiy6j+gdGYa4j4J+oh/uFWrVMXMZ3YF3g1aaSV3ImnYS9s37aCv7kbzJc0TH
CK9qezA+0tkP2gqzhmZZk/XuvtiLMOAd2FI41l22o63RaEF4k4H7jiEjATMq0zPN
nBlHb0yJBoHLjbBJervUBQ4184p1MGYZR/1BAsx7EWJ7Dj5QWOLCLykCAwEAAaOC
AeswggHnMB8GA1UdIwQYMBaAFJCvajqUWgvYkOoSVnPfQ7Q6KNrnMB0GA1UdDgQW
BBQ1KAcDJ0ddMekt+zdFLFtKv8sLmzAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/
BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwTwYDVR0gBEgwRjA6
BgsrBgEEAbIxAQICBzArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21v
ZG8uY29tL0NQUzAIBgZngQwBAgEwVAYDVR0fBE0wSzBJoEegRYZDaHR0cDovL2Ny
bC5jb21vZG9jYS5jb20vQ09NT0RPUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNl
cnZlckNBLmNybDCBhQYIKwYBBQUHAQEEeTB3ME8GCCsGAQUFBzAChkNodHRwOi8v
Y3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FEb21haW5WYWxpZGF0aW9uU2VjdXJl
U2VydmVyQ0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5j
b20wOQYDVR0RBDIwMIIXKi5wbHVzb25ldGVjaG5vbG9neS5uZXSCFXBsdXNvbmV0
ZWNobm9sb2d5Lm5ldDANBgkqhkiG9w0BAQsFAAOCAQEAOdjjTf0NzrbmQnmClML8
kPggIJhetlJzUaRMKV49oOWeRHyxF78DvTwelK+hCz1Of0nHddKTfUgcE1rfyGwB
hh0XtVyU32Kw8HNwb8KkJMCxFZ8A0mFlNXO5H1gHiJaZaUjHTJOR7RBJWfgxNrBb
3Wg/lOp55tl9fqDL29IvRqCldJl5arcJGz71Y6GUcIokUfp00JItvbj6AbLD+Wya
nXN6kATxGtigneZYCsiYlFVH5Qina3wjEyXR+L6wiPYBIYbQ4APQIR4R2gsooDw4
yja5CvKDuHk9/5jOQHiWD8CpBS5YPvuufuROzBgmDeoQx0TrwNClHynYIpD1RqE0
kA==
-----END CERTIFICATE-----
...

For tls.key:

jims@tupperware:~/flow/deis/certs$ cp star_plusonetechnology_net.key tls.crt

To generate the deis-router-platform-cert.yaml file, first the base64 encodings of tls.crt and tls.key are needed:

jims@tupperware:~/flow/deis/certs$ cat tls.crt | base64 > tls.crt.b64
jims@tupperware:~/flow/deis/certs$ cat tls.key | base64 > tls.key.b64
jims@tupperware:~/flow/deis/certs$ ls -l *b64
-rw-rw-r-- 1 jims jims 10233 Dec 25 17:06 tls.crt.b64
-rw-rw-r-- 1 jims jims  2266 Dec 25 17:06 tls.key.b64

Next, you need to create the deis-router-platform-cert.yaml in your editor of choice:

$ cat deis-router-platform-cert.yaml
apiVersion: v1
kind: Secret
metadata:
  name: deis-router-platform-cert
  namespace: deis
type: Opaque
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1...S0KDQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0...LS0tCg==

Save the file and use kubectl to add the new secret for the Deis Router:

jims@tupperware:~/flow/deis/certs$ kubectl create -f deis-router-platform-cert.yaml
secret "deis-router-platform-cert" created

You can see the change picked up by using kubectl to look at the logs for the Deis Router:

jims@tupperware:~$ kubectl --namespace=deis logs deis-router-1741606082-4mkzh
2016/12/26 01:13:58 INFO: Router configuration has changed in k8s.
2016/12/26 01:13:58 INFO: Reloading nginx...
2016/12/26 01:13:58 INFO: nginx reloaded.

To verify TLS is active for Deis Workflow, it is best to install a sample application and verify that it is reachable by both HTTP and HTTPS. The Deis Workflow Quickstart shows an example of a sample app being deployed, which will be used and then curl will be used to test both HTTP and HTTPS connections:

jims@tupperware:~$ deis create --no-remote
Creating Application... done, created united-greenery
If you want to add a git remote for this app later, use `deis git:remote -a united-greenery`
jims@tupperware:~/flow/deis/certs$ deis pull deis/example-go -a united-greenery
Creating build... done
jims@tupperware:~/flow/deis/certs$ curl http://united-greenery.plusonetechnology.net
Powered by Deis
jims@tupperware:~/flow/deis/certs$ curl https://united-greenery.plusonetechnology.net
Powered by Deis

Conclusion

Deis Workflow enables application developers to easily deploy applications based on the Twelve Factor model on Kubernetes clusters.  With Kubernetes available on the Azure Container Service, developers can leverage the resources of Azure in developing and deploying their applications.  With just a few additional steps, one can move deployed services from a test/dev environment to a production ready one.

In April of 2017, Microsoft acquired Deis.  This brings even more Kubernetes expertise to Microsoft and will make the overall experience of Deis (and Kubernetes) on Azure even better.

Category
ContainersDevOps
Topics
AzureAzure Cloud ComputingAzure Resource Manager (ARM)DeisKubernetesProduction

Author

James Spring

Bit wrangler by trade. Frequently spotted in the outdoors - rivers, forests, ocean, mountains. Know a little bit about California and Western States history, always learning more. Member of the fraternal order preserving Western history -- E Clampus Vitus

0 comments

Discussion are closed.

Read next