Who is Graph Story?
Graph Story is a startup based out of Memphis, Tennessee that offers fully-managed Neo4j databases as a service. Neo4j is one of the most popular graph database implementations, providing rich features to support machine learning and intelligence through relationships. Rather than users hosting and maintaining a Neo4j database themselves, they can leverage the Graph Story service and avoid the hassle of database management.
Enabling Azure Support for Graph Story
Knomos is a startup that is part of the Microsoft Accelerator winter 2016 class. They wanted to use Graph Story instead of hosting their own Neo4j server. However, they wanted their Graph Story managed database to run within Azure.
When we first worked with Graph Story, their database service had a variety of cloud vendors, but their Azure support was lacking. Graph Story, Knomos and Microsoft got together for one week in Seattle, Washington, to enable Graph Story’s database-as-a-service (DBaaS) on Azure.
Our goal was to make Graph Story’s automated database deployments to Azure consistent with the way they deploy to Digital Ocean, AWS and Google Compute Engine. Graph Story leverages Ansible, an application provisioning and deployment platform. With Ansible, DevOps engineers can provision and deploy apps similarly across different cloud platforms. In addition to working with Graph Story and Knomos, we also contributed to the Ansible project to enable the scenarios required by Graph Story.
Graph Story Deployment Strategy
Custom Image Creation & Deployment
Graph Story has a custom virtual machine image they use for their database nodes. In order for the image to work globally, they need to deploy each image to every Azure region they intend on supporting. The graphic below depicts how this works:
First a Gearman worker picks up the request for a new image to be created. The Ansible server is triggered via SSH to execute an Ansible playbook which uses 3 modules:
- Azure-Deploy: Deploys Azure templates defined by YAML within the playbook. In our case, it’s to provision the Target VM.
- Azure-Image-Capture: A module which captures an Azure VM image and stores the image as a blob.
- Azure-Blob-Copy: Copies a blob from one storage account to another. For Graph Story’s scenario, it’s called iteratively to copy the captured image to a list of storage accounts in different Azure regions.
Automated Graph Story Database Instances
When a customer purchases a database on GraphStory.com, the Graph Story web server sends a message to a Gearman worker. This triggers another Ansible playbook to provision a Graph Story instance onto Azure:
This is a similar architecture to the image deployment process; however, since the image already exists in the region targeted by the user, the deployment is much faster, providing a better customer experience.
Ansible Modules
We also collaborated with Ansible maintainers to publish the modules used by Graph Story for any Ansible user to capture and deploy Azure virtual machines using playbooks.
How to Deploy an Azure Virtual Machine using Ansible
Using the azure-deploy Ansible module, you can launch any sort of Azure Resource Template directly from your playbook YAML. Resource templates allow you to define a desired-state of your cloud infrastructure with a single deployment command. Typically, these templates are in JSON but with this module, they can be written in YAML. Here’s an example of deploying a virtual machine with the necessary storage and virtual network creation:
- name: Capture a VM image and copy it to other storage accounts
hosts: 127.0.0.1
connection: local
remote_user: ""
tasks:
- name: Create Azure VM
local_action:
module: azure_deploy
state: present
subscription_id: ""
client_id: ""
tenant_or_domain: ""
client_secret: "" # don't check in!
resource_group_name: ''
template:
$schema: "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#"
contentVersion: "1.0.0.0"
parameters:
adminUsername:
type: "string"
metadata:
description: "User name for the Virtual Machine."
dnsNameForPublicIP:
type: "string"
metadata:
description: "Unique DNS Name for the Public IP used to access the Virtual Machine."
ubuntuOSVersion:
type: "string"
defaultValue: "14.04.2-LTS"
allowedValues:
- "12.04.5-LTS"
- "14.04.2-LTS"
- "15.04"
metadata:
description: "The Ubuntu version for the VM. This will pick a fully patched image of this given Ubuntu version. Allowed values: 12.04.5-LTS, 14.04.2-LTS, 15.04."
variables:
newStorageAccountName: "[uniqueString(resourceGroup().id)]"
location: "West US"
imagePublisher: "Canonical"
imageOffer: "UbuntuServer"
OSDiskName: "osdiskforlinuxsimple"
nicName: "myVMNic"
addressPrefix: "10.0.0.0/16"
subnetName: "Subnet"
subnetPrefix: "10.0.0.0/24"
storageAccountType: "Standard_LRS"
publicIPAddressName: "myPublicIP"
publicIPAddressType: "Dynamic"
vmStorageAccountContainerName: "vhds"
vmName: ""
vmSize: "Standard_D1"
virtualNetworkName: "MyVNET"
vnetID: "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]"
subnetRef: "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]"
sshKeyPath: "[concat('/home/',parameters('adminUsername'),'/.ssh/authorized_keys')]"
resources:
-
type: "Microsoft.Storage/storageAccounts"
name: "[uniqueString(resourceGroup().id)]"
apiVersion: "2015-05-01-preview"
location: "[variables('location')]"
properties:
accountType: "[variables('storageAccountType')]"
-
apiVersion: "2015-05-01-preview"
type: "Microsoft.Network/publicIPAddresses"
name: "[variables('publicIPAddressName')]"
location: "[variables('location')]"
properties:
publicIPAllocationMethod: "[variables('publicIPAddressType')]"
dnsSettings:
domainNameLabel: "[parameters('dnsNameForPublicIP')]"
-
type: "Microsoft.Network/virtualNetworks"
apiVersion: "2015-05-01-preview"
name: "[variables('virtualNetworkName')]"
location: "[variables('location')]"
properties:
addressSpace:
addressPrefixes:
- "[variables('addressPrefix')]"
subnets:
-
name: "[variables('subnetName')]"
properties:
addressPrefix: "[variables('subnetPrefix')]"
-
type: "Microsoft.Network/networkInterfaces"
apiVersion: "2015-05-01-preview"
name: "[variables('nicName')]"
location: "[variables('location')]"
dependsOn:
- "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"
- "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
properties:
ipConfigurations:
-
name: "ipconfig1"
properties:
privateIPAllocationMethod: "Dynamic"
publicIPAddress:
id: "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
subnet:
id: "[variables('subnetRef')]"
-
type: "Microsoft.Compute/virtualMachines"
apiVersion: "2015-06-15"
name: "[variables('vmName')]"
location: "[variables('location')]"
dependsOn:
- "[concat('Microsoft.Storage/storageAccounts/', variables('newStorageAccountName'))]"
- "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
properties:
hardwareProfile:
vmSize: "[variables('vmSize')]"
osProfile:
computername: "[variables('vmName')]"
adminUsername: "[parameters('adminUsername')]"
adminPassword: "notused"
linuxConfiguration:
disablePasswordAuthentication: "true"
ssh:
publicKeys:
-
path: "[variables('sshKeyPath')]"
keyData: ""
storageProfile:
imageReference:
publisher: "[variables('imagePublisher')]"
offer: "[variables('imageOffer')]"
sku: "[parameters('ubuntuOSVersion')]"
version: "latest"
osDisk:
name: "osdisk"
vhd:
uri: "[concat('http://',variables('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
caching: "ReadWrite"
createOption: "FromImage"
networkProfile:
networkInterfaces:
-
id: "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
diagnosticsProfile:
bootDiagnostics:
enabled: "true"
storageUri: "[concat('http://',variables('newStorageAccountName'),'.blob.core.windows.net')]"
outputs:
storage_key:
type: "object"
value: "[listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('newStorageAccountName')), '2015-05-01-preview')]"
storage_name:
type: "string"
value: "[uniqueString(resourceGroup().id)]"
parameters:
adminUsername:
value: ''
dnsNameForPublicIP:
value: ''
register: azure
At the end of this module register azure will register the output of the deployment as a variable for use later in the playbook. In the example above, the outputs section of the template is returned, allowing us to extract useful information such as the storage account name and key.
Next, we’ll use the information from the Azure deployment module to add the newly created Azure virtual machine to our Ansible dynamic inventory.
- name: Add new instance to host group
add_host: hostname= groupname=launched ansible_user= vm_name=
with_items: azure.instances
azure.instances is a collection of Virtual Machine instances and this Ansible play will be called for each instance (using the with-items property), adding a new host for each iteration. Now you can provision or do anything else to this host as you normally would in Ansible, assuming that your SSH private key for the machine is accessible. We’ll add the new virtual machine to the launched inventory group for later use.
Graph Story runs their existing playbook steps used on all cloud providers after this step.
How to Capture an Azure Virtual Machine Image
Using the azure-image-capture module, capturing a virtual machine image becomes simple. First, you need to be sure to de-provision the user information generated by Azure within the target VM in order for it to be ready to become a ‘generic’ virtual machine. This ensures the image can be reused for creating any VM regardless of the local user settings. You can do this by running sudo waagent -deprovision+user which is a command to the Azure Linux agent:
- name: Run Deprovision user Command
hosts: launched
remote_user: ""
tasks:
- name: Deprovision User
command: sudo waagent -deprovision+user -force
Now the target virtual machine is ready to be captured:
- name: Capture Image
hosts: 127.0.0.1
connection: local
tasks:
- name: Capture
local_action:
module: azure_image_capture
subscription_id: ""
resource_group_name: ''
destination_container: copiedvhds
client_id: ""
tenant_id: ""
client_secret: ""
vm_name: ""
wait: ""
register: capture_info
After this step, the VM image has been captured, and a link to the image defined by capture_info.vhd_uri is returned. You can use this image directly to create a new VM, if it resides in the data center region you require. If you’re like Graph Story, you’ll need to copy the image to various region in order to use the image within regions other than where the generic VM was created.
We wrote the azure-copy-blob module specifically for this purpose:
- name: Copy Blob
local_action:
module: azure_copy_blob
source_uri: ""
source_key: ""
destination_account: ""
destination_container: ""
destination_key: ""
destination_blob: ""
with_items:
In the snippet above, we can use the output of the azure-image-capture module to copy the image from the source storage account to the destination. We can iterate using the with_items property by providing a collection of storage_accounts like this:
- vars:
storage_accounts:
-
account_name: graphstory4
key: STORAGE_ACCOUNT_KEY
container: storedimages
blob: graphstoryneo4j2.vhd
-
account_name: graphstory5
key: STORAGE_ACCOUNT_KEY
container: storedimages
blob: graphstoryneo4j2.vhd
-
account_name: graphstory6
key: STORAGE_ACCOUNT_KEY
container: storedimages
blob: graphstoryneo4j2.vhd
Graph Story uses this approach to deploy and update their Graph Story images across Azure. Checkout the Ansible Modules repository, where you can find these modules plus a great variety of others to use in your next automation project.
To get started with Graph Story, checkout their pricing page where you can select a Neo4j database which fits your needs.
0 comments