A Production Deis Workflow Deployment on Azure
Background
Deis Workflow is the second version of the Deis application platform that allows for the easy deployment and scalability of Twelve-Factor applications. Previously, we worked with Deis to enable their V1 PaaS on Azure based on CoreOS.
With Deis Workflow, the only requirement for installation is supporting Kubernetes. In November of 2016, the Azure Container Service announced support for Kubernetes on Azure. With this addition, we arranged another hackfest with the team from Deis.
The primary goal of the hackfest was to run the Deis Workflow End-to-End tests successfully. We accomplished this goal within the first few hours of the three-day event, which allowed time to explore other topics, including one that a customer was interested in: what a “production Deis Workflow deployment on Azure” would look like and the steps necessary to get there.
Deis Workflow, in addition to providing a platform for Twelve-Factor applications, simplifies working with Kubernetes. This allows developers to more readily focus on their apps initially without having to learn how to operate Kubernetes proper. As needed, or as familiarity improves, operating directly with Kubernetes is possible. Workflow and Kubernetes can be used without interference between the two.
The customer was interested in taking their current use of Kubernetes and Workflow for internal development and testing, and see what would be needed to move that setup into production. The definition of “production” in this context of Azure will rely on Deis’ documentation of a production deployment. The key points are:
- Internal state is moved to Azure Blob Store
- Update registration mode
- Enable TLS using a Platform certificate
Below we will cover the steps necessary get Deis Workflow up and running on Azure. We will also address the production deployment steps mentioned above in depth.
Requirements
To set up, configure, and install Deis Workflow, we will need to install a set of tools:
- Azure CLI: allows for interaction with various Azure services from the CLI.
- kubectl: primary tool for interacting with a Kubernetes cluster.
- Deis Workflow CLI: tool for interacting with Deis Workflow.
- Helm: Kubernetes package manager.
During the hackfest, the Azure Container Service had a certificate issue (since resolved) that required we use one additional tool:
- acs-engine: the open source engine for creating Azure Resource Manager (ARM) templates for deploying clusters (Docker Swarm, Kubernetes, and DC/OS) onto Azure
To fulfill the recommendations within the Deis Production Deployments document, we will need to:
- Provision an Azure Storage account to move Workflow’s internal state into persistent storage
- Procure a wildcard certificate to enable TLS
Setup
Below, we outline the needed steps and pointers to getting the tooling set up.
Third Party Tools
Each of the following third party tools, as mentioned above, need to be installed:
To verify each is properly configured:
jims@tupperware:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
jims@tupperware:~$ deis --version v2.9.1
jims@tupperware:~$ helm version
Client: &version.Version{SemVer:"v2.1.2", GitCommit:"58e545f47002e36ca71ac5d1f7a987b56e1937b3", GitTreeState:"clean"}
Error: cannot connect to Tiller
Azure CLI
The Azure CLI will be used to setup the necessary infrastructure components within Azure. The different options for installing the Azure CLI are here. Once installed, we will need to log in, which is a two-step process. First, you issue the az login command and then visit the Device Login page and enter the token presented on the command line.
jims@tupperware:~$ az login
To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code BXSABJN4S to authenticate.
[
{
"cloudName": "AzureCloud",
"id": "ab12ec88-8e28-41ed-8537-5e17766001f5",
"isDefault": true,
"name": "Developers Research",
"state": "Enabled",
"tenantId": "cfd488bf-86f1-41af-91ab-2d7cd011db47",
"user": {
"name": "jaspring@microsoft.com",
"type": "user"
}
}
]
When you go to Device Login, you will be prompted to enter the code shown in the above step. You may also need to log into your Azure account, as well.
At this point, you are ready to use the Azure CLI.
acs-engine
As mentioned previously, during the hackfest the ACS resource provider generated a certificate that did not play well with Deis Workflow. The certificate contained a value in the Subject Alternative Name field that the HTTP library Workflow used did not like. So, acs-engine was used to provision the cluster.
The steps necessary to install acs-engine are here.
Deis Workflow can be installed on any Kubernetes cluster, so one can use the Azure Portal or the Azure CLI Azure Container Service commands to provision a Kubernetes cluster on Azure in place of using acs-engine.
Preparation
With the tooling in place, there is some additional preparation needed, which is detailed in the following sections. You’ll need to:
- Create a Resource Group
- Generate an SSH Key
- Create a Service Principal
- Determine the details of the Kubernetes cluster
- Create a storage account and retrieve its key for the preservation of Deis Workflow’s state
- Create a certificate to enable TLS on Deis Workflow.
Create a Resource Group
A Resource Group is necessary for all Azure Resource Manager resources. It requires a name and a region to which it is deployed. For this tutorial, we will use:
- Name: deisprodk8s
- Location: westus
jims@tupperware:~$ az group create --location="westus" --name="deisprodk8s"
{
"id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s",
"location": "westus",
"managedBy": null,
"name": "deisprodk8s",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null
}
Generate an SSH Key
An SSH Key is needed to access machines in the Kubernetes cluster. The generation of keys will vary from platform to platform (for example, for Ubuntu, one can look here). For this tutorial, we will create a 2048 bit RSA key and store it in the file ~/.ssh/id_k8s_rsa:
jims@tupperware:~$ ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_k8s_rsa -C jaspring@microsoft.com Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/jims/.ssh/id_k8s_rsa. Your public key has been saved in /home/jims/.ssh/id_k8s_rsa.pub. The key fingerprint is: SHA256:HW7ptbn212QsD/VTWcZRNVtrl9ckFjmzkYVbqSe9WG8 jaspring@microsoft.com The key's randomart image is: +---[RSA 2048]----+ | +X@| | .B+%| | . o&*| | o o o+==| | S = . =o=| | o . +ooE| | . o Bo| | .. +| | ..... | +----[SHA256]-----+
For the Kubernetes configuration, the public key will be needed:
jims@tupperware:~$ more ~/.ssh/id_k8s_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqM F7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDd wf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjm KqvvuL3gZaQ4QCHz7 jaspring@microsoft.com
Create a Service Principal
A Service Principal is an entity that is granted permissions to perform operations within an Azure subscription. Kubernetes uses a Service Principal to manipulate compute resources as needed/defined during interactions with the Kubernetes cluster.
To create a Service Principal, use the Azure CLI:
jims@tupperware:~$ az ad sp create-for-rbac
{
"appId": "ad349d5e-2b0e-49ce-beaa-407c68196774",
"name": "http://azure-cli-2016-12-23-16-35-16",
"password": "b77abcf0-2f83-4e83-bc1a-c2c7dac8d9be",
"tenant": "123488bf-86f1-41af-92cb-2d7cd011db47"
}
Kubernetes Cluster details
For the cluster, we need additional information, like:
- Resource group: this is the grouping under which all resources will be mapped in Azure
- Region: which Azure Data Center to create the cluster in
- DNS Prefix: a prefix used to name the public components of the Kubernetes cluster
- Agent Machine Count: how many agent machines are needed
- Agent Machine Type: what type of VMs to use for the Kubernetes cluster
- Administrator Name: the name of the administrative user account on cluster nodes
For this deployment, we will use the following values:
- Resource group: deisprodk8s
- Region: westus (West US)
- DNS Prefix: deisprod
- Agent Machine Count: 4
- Agent Machine Type: Standard_D2_v2 (a general compute-oriented VM with 7GB RAM and 100GB local disk)
- Administrator Name: dadmin
acs-engine uses a configuration file to generate the templates necessary for deployment. Using the above information, including the Service Principal and the SSH Key, the configuration file ~/flow/k8s/config/kubernetes.json resembles the following:
bash
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "deisprod",
"vmSize": "Standard_D2_v2"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 4,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet"
}
],
"linuxProfile": {
"adminUsername": "dadmin",
"ssh": {
"publicKeys": [
{
"keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com"
}
]
}
},
"servicePrincipalProfile": {
"servicePrincipalClientID": "ad349d5e-2b0e-49ce-beaa-407c68196774",
"servicePrincipalClientSecret": "b77abcf0-2f83-4e83-bc1a-c2c7dac8d9be"
}
}
}
Create Storage Account and Retrieve Key
The storage account will be created in the same Resource Group as defined above, deisprodk8s. The settings we will use for the storage account are:
- Storage Account Name: deisprodconfig
- Storage Account SKU: Standard_LRS (Standard, Locally Redundant Storage)
- Storage Account Kind: Storage
Creating the storage account:
jims@tupperware:~$ az storage account create --sku="Standard_LRS" --resource-group="deisprodk8s" --kind="Storage" --location="westus" --name="deisprodconfig"
{
"accessTier": null,
"creationTime": "2016-12-23T17:10:32.334380+00:00",
"customDomain": null,
"encryption": null,
"id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Storage/storageAccounts/deisprodconfig",
"kind": "Storage",
"lastGeoFailoverTime": null,
"location": "westus",
"name": "deisprodconfig",
"primaryEndpoints": {
"blob": "https://deisprodconfig.blob.core.windows.net/",
"file": "https://deisprodconfig.file.core.windows.net/",
"queue": "https://deisprodconfig.queue.core.windows.net/",
"table": "https://deisprodconfig.table.core.windows.net/"
},
"primaryLocation": "westus",
"provisioningState": "Succeeded",
"resourceGroup": "deisprodk8s",
"secondaryEndpoints": null,
"secondaryLocation": null,
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"statusOfPrimary": "Available",
"statusOfSecondary": null,
"tags": {},
"type": "Microsoft.Storage/storageAccounts"
}
Retrieving the storage account keys:
jims@tupperware:~$ az storage account keys list --name="deisprodconfig" --resource-group="deisprodk8s"
{
"keys": [
{
"keyName": "key1",
"permissions": "FULL",
"value": "f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ=="
},
{
"keyName": "key2",
"permissions": "FULL",
"value": "ad9pMFJUualKDH3irkNyS3XXH0xMbs0JC56ESwiVnRaCJUnuc/6FEQf09xzjsK4vO34LNq+4VV1N20DI9kau5g=="
}
]
}
Create a Certificate to Enable TLS
There are two approaches you can take to enable TLS on Deis Workflow. The first is to create an application certificate; for instance, if all of the public traffic will come through www.example.com, then only a certificate specific to www.example.com would be needed. These certificates are typically referred to as domain-specific certificates. Within Deis, these might be referred to as application-specific certificates.
A more general approach is to use what is referred to as a wildcard certificate. The domains specified in the wildcard certificate would resemble *.example.com. Thus any name prefixing .example.com would match. This certificate can be installed in Deis Workflow as a platform certificate.
For development and testing, you can use OpenSSL to generate a self-signed certificate. Digital Ocean has a good tutorial. Another option is to consider is Let’s Encrypt.
For production, certificates typically cost money. Their retrieval can be quick or take a while, depending on the type of certificate used.
In this tutorial, we will request a PositiveSSL Wildcard certificate from Namecheap.
To get the certificate issued, you must create a Certificate Signing Request (CSR). Certificates have several fields that can be specified for the CSR. In this case, we will specify the following:
- Country (C): US
- State (ST): California
- Locality (L): Santa Cruz
- Organization (O): plusonetechnology.net
- Common Name (CN): *.plusonetechnology.net
The Common Name field is what a TLS connection is verified against when a connection comes into the server.
To create the CSR:
jims@tupperware:~$ openssl req -new -newkey rsa:2048 -nodes > -out star_plusonetechnology_net.csr > -keyout star_plusonetechnology_net.key > -subj "/C=US/ST=California/L=Santa Cruz/O=plusonetechnology.net/CN=*.plusonetechnology.net" Generating a 2048 bit RSA private key ................................+++ .....+++ writing new private key to 'star_plusonetechnology_net.key' -----
The CSR is written as a PEM blob; in this case, it is written to the file star_plusonetechnology_net.csr (contents shown below):
jims@tupperware:~$ more star_plusonetechnology_net.csr -----BEGIN CERTIFICATE REQUEST----- MIICvjCCAaYCAQAweTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWEx EzARBgNVBAcTClNhbnRhIENydXoxHjAcBgNVBAoTFXBsdXNvbmV0ZWNobm9sb2d5 Lm5ldDEgMB4GA1UEAxQXKi5wbHVzb25ldGVjaG5vbG9neS5uZXQwggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDoudQXqW5++L5a1fUtBNNlXuLeUOJII+Ci 10MdWWUosyn1hHku0AcBXHdONDLfq2W1jUMrxlXaDWKSDaM1xLUCVVKfiK9tpM/v erhVhddUATHE8ZSXPY/HZGkCCoJlF88FDM+aKieteQn6mN7EjemvhtNB2suy455S WKosuo/oHRmGuI+CfqIf7hVq1TFzGd2Bd4NWmkldyJp2EvbN+2gr+5G8yXNExwiv answPtLZD9oKs4ZmWZP17r7YizDgHdhSONZdtqOt0WhBeJOB+44hIwEzKtMzzZwZ R29MiQaBy42wSXq71AUONfOKdTBmGUf9QQLMexFiew4+UFjiwi8pAgMBAAGgADAN BgkqhkiG9w0BAQUFAAOCAQEAcyuepBBHSHNh4zlCIhHcnQnTni1COHMx/7DIaeTq UCJm4Yt26n77EwdK/b4l97l09PIBOdZowf9ETSgLvd2jy7KmbMFQMuRZmH1jZwSa Cj8DV8PixA4qm9ARBAugQmwoEFVPmxWCC+lD7O8ti+bolP6twS/dsjM3bPCu1l23 5X8pqHfrYPI/C28INKdr2TdUFQi0ocFIqm3RkYhfGjOrTii9m99enf3HFFX9Pr8z p1QjSlewyOrkiXcQIAP1Fnck48txFCyU5tFsyTBoF6y91jR9GY5wL/kDwvxbFiWq dBY3knjwtaKe/iLPL7YV2NNUA/+dJ014NJW3oeJRnnfQlQ== -----END CERTIFICATE REQUEST-----
For issuing the certificate, Namecheap will require that the CSR be pasted into the website. It will also require some personal information and a process to validate that the domain specified in the certificate is owned by the person submitting the request. Since this process will depend on where the certificate is purchased and the CA used, the sequence will be skipped.
For the CSR for plusonetechnology.net, via Namecheap, DNS-based validation was used.
Provisioning the Kubernetes Cluster
Previously, we defined and updated the acs-engine configuration file as ~/flow/k8s/config/kubernetes.json. When acs-engine is run against this file, it will generate the necessary Azure Resource Manager (ARM) templates for deploying the Kubernetes cluster:
jims@tupperware:~/flow/k8s/config$ acs-engine -artifacts ~/flow/k8s/output ~/flow/k8s/config/kubernetes.json cert creation took 7.712984265s wrote /home/jims/flow/k8s/output/apimodel.json wrote /home/jims/flow/k8s/output/azuredeploy.json wrote /home/jims/flow/k8s/output/azuredeploy.parameters.json ... wrote /home/jims/flow/k8s/output/kubeconfig/kubeconfig.westus.json ... acsengine took 7.743813684s
Note: the ellipses (…) imply content that has been cut to keep the output shown more concise.
The next step is to deploy the templates. Again, the Resource Group we are using is deisprodk8s.
jims@tupperware:~/flow/k8s/config$ az group deployment create --resource-group="deisprodk8s" --template-file=/home/jims/flow/k8s/output/azuredeploy.json --parameters @/home/jims/flow/k8s/output/azuredeploy.parameters.json
{
"id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Resources/deployments/azuredeploy",
"name": "azuredeploy",
"properties": {
"correlationId": "a58352c4-b97d-4e89-bf23-e10e0c9637da",
"debugSetting": null,
"dependencies": [
...
],
"mode": "Incremental",
"outputs": {
"masterFQDN": {
"type": "String",
"value": "deisprod.westus.cloudapp.azure.com"
}
},
"parameters": {
...
},
"parametersLink": null,
"providers": [
...
],
"provisioningState": "Succeeded",
"template": null,
"templateLink": null,
"timestamp": "2016-12-23T21:54:04.178783+00:00"
},
"resourceGroup": "deisprodk8s"
}
Setting Up `kubectl`
With the Kubernetes cluster deployed into Azure, we can now configure settings so that kubectl can talk to the cluster and we can interact with things.
When acs-engine was run, a number of kubeconfig.*.json files were generated for each region. Since our example deployment is in westus, the one we will look at is kubeconfig.westus.json. The KUBECONFIG environment variable needs to be set and its status checked to utilize this file, as shown below:
jims@tupperware:~/flow/k8s/output$ export KUBECONFIG=/home/jims/flow/k8s/output/kubeconfig/kubeconfig.westus.json jims@tupperware:~/flow/k8s/output$ kubectl cluster-info Kubernetes master is running at https://deisprod.westus.cloudapp.azure.com Heapster is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster KubeDNS is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at https://deisprod.westus.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
To further debug and diagnose cluster problems, use kubectl cluster-info dump.
At this point, kubectl is configured and we can interact with the cluster. Please note that adding the export KUBECONFIG to one’s startup files will be helpful for subsequent logins to the machine from which one would interact with the cluster. Â You can also copy the configuration file to ${HOME}/.kube/config.
Kubernetes Using the `az` Command line
A much simpler way to install the Kubernetes cluster from the command line is available. For completeness, the method to deploy the same Kubernetes cluster is documented below. The assumption is that the configuration is the same as what we specified in the kubernetes.json file used by acs-engine. That is:
- Resource group: deisprodk8s
- Region: westus
- DNS Prefix: deisprod
- Agent Machine Count: 4
- Agent Machine Type: Standard_D2_v2
- Administrator Name: dadmin
-
SSH Key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com
jims@tupperware:~$ az acs create --name="deisprod"
> --orchestrator-type="Kubernetes"
> --resource-group="deisprodk8s"
> --admin-username="dadmin"
> --agent-count=4
> --agent-vm-size="Standard_D2_v2"
> --dns-prefix="deisprod"
> --location="westus"
> --ssh-key-value="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXlnm3BkYMPoXUd+r3BKmF8+k+3tVudOLviZK63FtOETAk6u4q3hk35fs/YfIyF7BTf+f3cQ3FQwN9lOZqMF7WyfI+ncgtrdsD4CqMB9ewQRYmrnQMxO0QinfEUi2G6qRXTMZlaPRYXJo4MAklU6k9b9XPrzoodkB3eoxh/XIah5oHUYmKlSNfTdy1xDjxhOwHoJ7BiqNKDdwf1pLwRpPW7Vg8OirqoX11CNgCPNeBclwxF3Vxzj4MNOUb18XvLXM8IGoGzOSaMhLKbK9pnUA/iMto4WabaBOnf6c9ADmK6gndVUU3lkALnIRgWsEsTfa9rjmKqvvuL3gZaQ4QCHz7 jaspring@microsoft.com"
creating service principal.........done
waiting for AAD role to propogate.done
{
"id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Resources/deployments/azurecli1482540765.0587363",
"name": "azurecli1482540765.0587363",
"properties": {
"correlationId": "b1a52fec-8b35-49e7-af8a-e0d7ad9d056e",
"debugSetting": null,
"dependencies": [],
"mode": "Incremental",
"outputs": null,
"parameters": {
"clientSecret": {
"type": "SecureString"
}
},
"parametersLink": null,
"providers": [
{
"id": null,
"namespace": "Microsoft.ContainerService",
"registrationState": null,
"resourceTypes": [
{
"aliases": null,
"apiVersions": null,
"locations": [
"westus"
],
"properties": null,
"resourceType": "containerServices"
}
]
}
],
"provisioningState": "Succeeded",
"template": null,
"templateLink": null,
"timestamp": "2016-12-24T00:59:18.590702+00:00"
},
"resourceGroup": "deisprodk8s"
}
Installing Deis Workflow
Previously, both the Deis CLI and Helm were installed. With the KUBECONFIG variable set in the prior section, the steps needed to install Deis Workflow are straightforward. The only potential issue is that we need to set up the configuration to reflect the production concerns previously mentioned, which include:
- Internal state/storage will use Azure Storage
- Registration for users should be admin_only
Installing Deis requires the following initial steps:
- Check your setup and initialize Helm
jims@tupperware:~/$ helm version Client: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"} Error: cannot connect to Tiller jims@tupperware:~/$ helm init $HELM_HOME has been configured at /home/jims/flow/helm. Tiller (the helm server side component) has been installed into your Kubernetes Cluster. Happy Helming! jims@tupperware:~/$ helm version Client: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.3.0", GitCommit:"d83c245fc324117885ed83afc90ac74afed271b4", GitTreeState:"clean"} - Add the Deis Workflow repository
jims@tupperware:~/$ helm repo add deis https://charts.deis.com/workflow "deis" has been added to your repositories
During the install of Workflow, in order to meet the requirements laid out for a production system, some configuration information is needed. Specifically, the information from the storage account we setup previously. In addition, we need to know what config options are needed to lock down user registration to the admin_only.
Note:Â Since the hackfest, Deis Workflow defaults to registration being admin_only.
To configure and install Workflow, we need:
- Storage Account Name:Â deisprodconfig
- Storage Account Key: f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ==
And install Workflow as follows:
$ export STORAGE_ACCOUNT_NAME=deisprodconfig $ export STORAGE_ACCOUNT_KEY="f8nHiACBlR71UgHjPuShSukge27c+bOty2M2TmTNy61zgPXlXAZuABLxJ/zG+3VHhvb18Ux5I9rx+1oQ3buaGQ==" $ helm install deis/workflow --namespace=deis > --set global.storage=azure,azure.accountname=$STORAGE_ACCOUNT_NAME,azure.accountkey=$STORAGE_ACCOUNT_KEY,azure.registry_container=registry,azure.database_container=database,azure.builder_container=builder NAME: whopping-ostrich LAST DEPLOYED: Wed May 10 12:16:06 2017 NAMESPACE: deis STATUS: DEPLOYED RESOURCES: ==> extensions/v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deis-monitor-grafana 1 1 1 0 3s deis-logger-redis 1 1 1 0 3s deis-database 1 1 1 0 3s deis-controller 1 1 1 0 3s deis-registry 1 1 1 0 3s deis-monitor-influxdb 1 1 1 0 3s deis-nsqd 1 1 1 0 3s deis-logger 1 1 1 0 3s deis-router 1 1 1 0 3s deis-workflow-manager 1 1 1 0 3s deis-builder 1 1 1 0 3s ==> v1/Secret NAME TYPE DATA AGE deis-router-dhparam Opaque 1 4s minio-user Opaque 2 4s objectstorage-keyfile Opaque 5 4s ==> v1/ConfigMap NAME DATA AGE slugrunner-config 1 4s slugbuilder-config 2 4s dockerbuilder-config 2 4s ==> v1/ServiceAccount NAME SECRETS AGE deis-logger 1 4s deis-router 1 4s deis-builder 1 4s deis-controller 1 4s deis-registry 1 4s deis-monitor-telegraf 1 4s deis-nsqd 1 4s deis-workflow-manager 1 4s deis-logger-fluentd 1 4s deis-database 1 4s ==> v1/Service NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE deis-controller 10.0.83.212 80/TCP 4s deis-registry 10.0.93.44 80/TCP 4s deis-nsqd 10.0.6.2 4151/TCP,4150/TCP 4s deis-router 10.0.128.55 80:30854/TCP,443:30567/TCP,2222:31387/TCP,9090:32093/TCP 4s deis-logger 10.0.55.43 80/TCP 4s deis-builder 10.0.206.67 2222/TCP 4s deis-workflow-manager 10.0.182.215 80/TCP 4s deis-monitor-grafana 10.0.39.43 80/TCP 3s deis-monitor-influxui 10.0.236.49 80/TCP 3s deis-database 10.0.184.196 5432/TCP 3s deis-monitor-influxapi 10.0.214.212 80/TCP 3s deis-logger-redis 10.0.146.66 6379/TCP 3s ==> extensions/v1beta1/DaemonSet NAME DESIRED CURRENT READY NODE-SELECTOR AGE deis-monitor-telegraf 5 5 0 3s deis-logger-fluentd 5 5 0 3s deis-registry-proxy 5 5 0 3s
To verify the Deis Workflow deployment is complete, you can use the command kubectl –namespace=deis get pods. When installation is complete, output should resemble:
jims@tupperware:~$ kubectl --namespace=deis get pods NAME READY STATUS RESTARTS AGE deis-builder-587185222-rktvr 1/1 Running 0 6m deis-controller-1274199582-3lqgv 1/1 Running 0 6m deis-database-1755707874-9z55k 1/1 Running 0 6m deis-logger-2533678197-91h6m 1/1 Running 0 6m deis-logger-fluentd-69dqb 1/1 Running 4 6m deis-logger-fluentd-bw2v7 1/1 Running 0 6m deis-logger-fluentd-j5zt9 1/1 Running 0 6m deis-logger-fluentd-mlddn 1/1 Running 0 6m deis-logger-fluentd-sh8kw 1/1 Running 0 6m deis-logger-redis-1307646428-w32k3 1/1 Running 0 6m deis-monitor-grafana-59098797-2v6p2 1/1 Running 0 6m deis-monitor-influxdb-168332144-5l7bx 1/1 Running 0 6m deis-monitor-telegraf-624n4 1/1 Running 0 6m deis-monitor-telegraf-6bmd6 1/1 Running 0 6m deis-monitor-telegraf-bk833 1/1 Running 1 6m deis-monitor-telegraf-x98sw 1/1 Running 0 6m deis-monitor-telegraf-zzjjl 1/1 Running 0 6m deis-nsqd-1042535208-b6th3 1/1 Running 0 6m deis-registry-3958274866-qjv9h 1/1 Running 0 6m deis-registry-proxy-58tv0 1/1 Running 0 6m deis-registry-proxy-86rlw 1/1 Running 0 6m deis-registry-proxy-9h3cz 1/1 Running 0 6m deis-registry-proxy-djrts 1/1 Running 0 6m deis-registry-proxy-mtc7d 1/1 Running 0 6m deis-router-3258454730-5l05p 1/1 Running 0 6m deis-workflow-manager-3582051402-1tzl6 1/1 Running 0 6m
Finalizing Deis Workflow Readiness
At this point, Deis Workflow is up and running on Kubernetes on Azure. However, we still have a few steps left to finalize the readiness of the deployment, including:
- Setting up DNS for Deis Workflow
- Registering the Deis Workflow Administrator
- Installing the Deis Platform Certificate and Enable TLS
Setting up DNS for Deis Workflow
When applications are deployed to Workflow, a name can be specified, or one will be generated. That name is prefixed to the domain set up for Deis Workflow. Previously, a wildcard certificate was registered for the domain plusonetechnology.net. Azure DNS needs to be configured to point to Deis Workflow. We must do the following steps, detailed in the following sections:
- Configure Azure DNS Zone for the domain name
- Update DNS Entries for the domain name at the domain registrar
- Test to verify DNS has updated
Configure Azure DNS
To configure Azure DNS you must:
- Create the Azure DNS Zone
- Create the Wildcard A Record-set
- Create the Wildcard A Record
To create the zone:
jims@tupperware:~$ az network dns zone create --name="plusonetechnology.net" --resource-group="deisprodk8s"
{
"etag": "00000002-0000-0000-3639-3f9f965dd201",
"id": "/subscriptions/ab12ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/deisprodk8s/providers/Microsoft.Network/dnszones/plusonetechnology.net",
"location": "global",
"maxNumberOfRecordSets": 5000,
"name": "plusonetechnology.net",
"nameServers": [
"ns1-07.azure-dns.com.",
"ns2-07.azure-dns.net.",
"ns3-07.azure-dns.org.",
"ns4-07.azure-dns.info."
],
"numberOfRecordSets": 2,
"resourceGroup": "deisprodk8s",
"tags": {},
"type": "Microsoft.Network/dnszones"
}
In the information returned, note the list of name servers. This list will be needed when updating the information for the domain at the domain registrar. This process will be covered in the next section.
Creating the record set:
jims@tupperware:~$ az network dns record-set a create
> --name="*"
> --zone-name="plusonetechnology.net"
> --resource-group="deisprodk8s"
{
"etag": "16fb3cb9-2a63-4ca2-babb-74e7a08fc9a6",
"id": "/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/jmsdeis/providers/Microsoft.Network/dnszones/plusonetechnology.net/A/*",
"metadata": null,
"name": "*",
"resourceGroup": "deisprodk8s",
"ttl": 3600,
"type": "Microsoft.Network/dnszones/A"
}
As part of the Deis Workflow setup, Deis provisions a load balancer for inbound traffic, referred to as the LoadBalancer Ingress, which is a public IP address. To set up the A-Record for the domain, we need to retrieve this address. Querying the Kubernetes cluster as shown below returns this information:
```bash jims@tupperware:~$ kubectl --namespace=deis describe svc deis-router Name: deis-router Namespace: deis Labels: heritage=deis Annotations: service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout=1200 Selector: app=deis-router Type: LoadBalancer IP: 10.0.128.55 LoadBalancer Ingress: 52.173.130.253 Port: http 80/TCP NodePort: http 30854/TCP Endpoints: 10.244.0.7:8080 Port: https 443/TCP NodePort: https 30567/TCP Endpoints: 10.244.0.7:6443 Port: builder 2222/TCP NodePort: builder 31387/TCP Endpoints: 10.244.0.7:2222 Port: healthz 9090/TCP NodePort: healthz 32093/TCP Endpoints: 10.244.0.7:9090 Session Affinity: None Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 15m 15m 1 service-controller Normal CreatingLoadBalancer Creating load balancer 12m 12m 1 service-controller Normal CreatedLoadBalancer Created load balancer
As you can see above, the LoadBalancer Ingress IP address is 52.173.130.253. This value is used to set up the A-Record for Azure DNS:
jims@tupperware:~$ az network dns record-set a add-record --record-set-name="*" --resource-group="deisprodk8s" --zone-name="plusonetechnology.net" --ipv4-address=52.173.130.253
{
"arecords": [
{
"ipv4Address": "52.173.130.253"
}
],
"etag": "99420e4d-8bfc-4b68-8509-1bd73f02c2ac",
"id": "/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5/resourceGroups/jmsdeis/providers/Microsoft.Network/dnszones/plusonetechnology.net/A/*",
"metadata": null,
"name": "*",
"resourceGroup": "jmsdeis",
"ttl": 3600,
"type": "Microsoft.Network/dnszones/A"
}
At this point, DNS is configured on the Azure side. The next step is to update the DNS information with the domain registrar.
Update DNS Entries for the Domain Name at the Domain Registrar
The method for updating DNS information will vary from registrar to registrar. The domain plusonetechnology.net is registered with Joker. Updating the DNS records for plusonetechnology.net at Joker will look like:
Note that the values in the image are the same nameserver values returned when we created the DNS Zone with the Azure CLI above.
Test to Verify DNS Has Updated
After changing that information on the domain registrar, we need to verify that the DNS has been updated. This process can take just a few minutes or it can be longer.
Since a Wildcard record was set up for the domain plusonetechnology.net, any name placed in from of the domain will now resolve to the same address.
jims@tupperware:~$ nslookup muppets.plusonetechnology.net Server: 10.211.55.1 Address: 10.211.55.1#53 Non-authoritative answer: Name: muppets.plusonetechnology.net Address: 52.173.130.25
At this point, DNS is properly configured, and the information has propagated.
Register the Deis Workflow Administrator
With a new Deis Workflow install, the first user registered is the administrator. To register:
jims@tupperware:~$ deis register http://deis.plusonetechnology.net username: jims password: password (confirm): email: jaspring@microsoft.com Registered jims Logged in as jims Configuration file written to /home/jims/.deis/client.json jims@tupperware:~$ deis users === Users (*=admin) *jims
Install the Deis Platform Certificate and Enable TLS
Previously, a Wildcard Certificate was ordered from Namecheap. After going through the validation process, the certificates were delivered:
jims@tupperware:__plusonetechnology_net$ ls -ltra total 40 drwx------@ 5 jims staff 170 Dec 23 13:24 . drwx------+ 15 jims staff 510 Dec 23 15:31 .. -rwxr-xr-x@ 1 jims staff 7342 Dec 23 16:24 __plusonetechnology_net.p7b -rwxr-xr-x@ 1 jims staff 1944 Dec 23 16:24 __plusonetechnology_net.crt -rwxr-xr-x@ 1 jims staff 5630 Dec 23 16:24 __plusonetechnology_net.ca-bundle
This step is in addition to the private key and certificate signing request generated earlier.
Deis Workflow specifies how to enable Platform SSL. In short, the steps are:
- Create a file tls.crt that has the TLS certificate, first, and chain certificates following
- Create a file tls.key which is the private key file copied/renamed
- Create a YAML file to update Deis Router settings
- Secret name is deis-router-platform-cert
- Requires both tls.crt and tls.key be base64 encoded
- Use kubectl to add the new configuration
For tls.crt:
jims@tupperware:~/flow/deis/certs$ cat __plusonetechnology_net.crt __plusonetechnology_net.ca-bundle > tls.crt jims@tupperware:~/flow/deis/certs$ more tls.crt -----BEGIN CERTIFICATE----- MIIFbTCCBFWgAwIBAgIRAJJIdJMChgmODD2ZZ/HKQQwwDQYJKoZIhvcNAQELBQAw gZAxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO BgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTYwNAYD VQQDEy1DT01PRE8gUlNBIERvbWFpbiBWYWxpZGF0aW9uIFNlY3VyZSBTZXJ2ZXIg Q0EwHhcNMTYxMjIzMDAwMDAwWhcNMTcxMjIzMjM1OTU5WjBkMSEwHwYDVQQLExhE b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxHTAbBgNVBAsTFFBvc2l0aXZlU1NMIFdp bGRjYXJkMSAwHgYDVQQDDBcqLnBsdXNvbmV0ZWNobm9sb2d5Lm5ldDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAOi51Bepbn74vlrV9S0E02Ve4t5Q4kgj 4KLXQx1ZZSizKfWEeS7QBwFcd040Mt+rZbWNQyvGVdoNYpINozXEtQJVUp+Ir22k z+96uFWF11QBMcTxlJc9j8dkaQIKgmUXzwUMz5oqJ615CfqY3sSN6a+G00Hay7Lj nlJYqiy6j+gdGYa4j4J+oh/uFWrVMXMZ3YF3g1aaSV3ImnYS9s37aCv7kbzJc0TH CK9qezA+0tkP2gqzhmZZk/XuvtiLMOAd2FI41l22o63RaEF4k4H7jiEjATMq0zPN nBlHb0yJBoHLjbBJervUBQ4184p1MGYZR/1BAsx7EWJ7Dj5QWOLCLykCAwEAAaOC AeswggHnMB8GA1UdIwQYMBaAFJCvajqUWgvYkOoSVnPfQ7Q6KNrnMB0GA1UdDgQW BBQ1KAcDJ0ddMekt+zdFLFtKv8sLmzAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/ BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwTwYDVR0gBEgwRjA6 BgsrBgEEAbIxAQICBzArMCkGCCsGAQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21v ZG8uY29tL0NQUzAIBgZngQwBAgEwVAYDVR0fBE0wSzBJoEegRYZDaHR0cDovL2Ny bC5jb21vZG9jYS5jb20vQ09NT0RPUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNl cnZlckNBLmNybDCBhQYIKwYBBQUHAQEEeTB3ME8GCCsGAQUFBzAChkNodHRwOi8v Y3J0LmNvbW9kb2NhLmNvbS9DT01PRE9SU0FEb21haW5WYWxpZGF0aW9uU2VjdXJl U2VydmVyQ0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9jYS5j b20wOQYDVR0RBDIwMIIXKi5wbHVzb25ldGVjaG5vbG9neS5uZXSCFXBsdXNvbmV0 ZWNobm9sb2d5Lm5ldDANBgkqhkiG9w0BAQsFAAOCAQEAOdjjTf0NzrbmQnmClML8 kPggIJhetlJzUaRMKV49oOWeRHyxF78DvTwelK+hCz1Of0nHddKTfUgcE1rfyGwB hh0XtVyU32Kw8HNwb8KkJMCxFZ8A0mFlNXO5H1gHiJaZaUjHTJOR7RBJWfgxNrBb 3Wg/lOp55tl9fqDL29IvRqCldJl5arcJGz71Y6GUcIokUfp00JItvbj6AbLD+Wya nXN6kATxGtigneZYCsiYlFVH5Qina3wjEyXR+L6wiPYBIYbQ4APQIR4R2gsooDw4 yja5CvKDuHk9/5jOQHiWD8CpBS5YPvuufuROzBgmDeoQx0TrwNClHynYIpD1RqE0 kA== -----END CERTIFICATE----- ...
For tls.key:
jims@tupperware:~/flow/deis/certs$ cp star_plusonetechnology_net.key tls.crt
To generate the deis-router-platform-cert.yaml file, first the base64 encodings of tls.crt and tls.key are needed:
jims@tupperware:~/flow/deis/certs$ cat tls.crt | base64 > tls.crt.b64 jims@tupperware:~/flow/deis/certs$ cat tls.key | base64 > tls.key.b64 jims@tupperware:~/flow/deis/certs$ ls -l *b64 -rw-rw-r-- 1 jims jims 10233 Dec 25 17:06 tls.crt.b64 -rw-rw-r-- 1 jims jims 2266 Dec 25 17:06 tls.key.b64
Next, you need to create the deis-router-platform-cert.yaml in your editor of choice:
$ cat deis-router-platform-cert.yaml apiVersion: v1 kind: Secret metadata: name: deis-router-platform-cert namespace: deis type: Opaque data: tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1...S0KDQo= tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0...LS0tCg==
Save the file and use kubectl to add the new secret for the Deis Router:
jims@tupperware:~/flow/deis/certs$ kubectl create -f deis-router-platform-cert.yaml secret "deis-router-platform-cert" created
You can see the change picked up by using kubectl to look at the logs for the Deis Router:
jims@tupperware:~$ kubectl --namespace=deis logs deis-router-1741606082-4mkzh 2016/12/26 01:13:58 INFO: Router configuration has changed in k8s. 2016/12/26 01:13:58 INFO: Reloading nginx... 2016/12/26 01:13:58 INFO: nginx reloaded.
To verify TLS is active for Deis Workflow, it is best to install a sample application and verify that it is reachable by both HTTP and HTTPS. The Deis Workflow Quickstart shows an example of a sample app being deployed, which will be used and then curl will be used to test both HTTP and HTTPS connections:
jims@tupperware:~$ deis create --no-remote Creating Application... done, created united-greenery If you want to add a git remote for this app later, use `deis git:remote -a united-greenery` jims@tupperware:~/flow/deis/certs$ deis pull deis/example-go -a united-greenery Creating build... done jims@tupperware:~/flow/deis/certs$ curl http://united-greenery.plusonetechnology.net Powered by Deis jims@tupperware:~/flow/deis/certs$ curl https://united-greenery.plusonetechnology.net Powered by Deis
Conclusion
Deis Workflow enables application developers to easily deploy applications based on the Twelve Factor model on Kubernetes clusters. Â With Kubernetes available on the Azure Container Service, developers can leverage the resources of Azure in developing and deploying their applications. Â With just a few additional steps, one can move deployed services from a test/dev environment to a production ready one.
In April of 2017, Microsoft acquired Deis. Â This brings even more Kubernetes expertise to Microsoft and will make the overall experience of Deis (and Kubernetes) on Azure even better.
Light
Dark
0 comments