{"id":2493,"date":"2017-04-18T15:55:36","date_gmt":"2017-04-18T15:55:36","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=2493"},"modified":"2020-03-15T05:34:00","modified_gmt":"2020-03-15T12:34:00","slug":"continuous-delivery-multi-cloud-marketplaces","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/continuous-delivery-multi-cloud-marketplaces\/","title":{"rendered":"Continuous Delivery to Multi-Cloud Marketplaces"},"content":{"rendered":"<p>Recently, we collaborated with <a href=\"https:\/\/www.noobaa.com\/\">NooBaa<\/a>,\u00a0a company that provides cloud-based data abstraction, including\u00a0a hybrid approach between on-premises and cloud-based storage. Their offerings support S3 and Lambda APIs that run using integrated Windows and Linux servers, Azure Blob Storage and AWS S3 resources within a single namespace.\u00a0Other partners we have worked with, including Jenkins, Cloudera, and MongoDB, have had similar requirements.<\/p>\n<p>Previously, NooBaa had four separate, semi-manual processes (one for each cloud platform) to execute the following pipeline:<\/p>\n<p><figure id=\"attachment_2784\" aria-labelledby=\"figcaption_attachment_2784\" class=\"wp-caption alignleft\" ><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/04\/Pipeline-1.png\" alt=\"Image Pipeline 1\" width=\"1247\" height=\"391\" class=\"aligncenter size-full wp-image-11020\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/Pipeline-1.png 1247w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/Pipeline-1-300x94.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/Pipeline-1-1024x321.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/Pipeline-1-768x241.png 768w\" sizes=\"(max-width: 1247px) 100vw, 1247px\" \/><figcaption id=\"figcaption_attachment_2784\" class=\"wp-caption-text\"><strong>Continuous Delivery Pipeline<\/strong><\/figcaption><\/figure><\/p>\n<p>These processes require one or more people to perform, as well as appropriate knowledge of each cloud platform on which the customer wishes to deploy their solution. The requirements for this pipeline, including the fact it contains manual tasks, turns it into\u00a0a bottleneck that prevents NooBaa from running\u00a0their pipeline\u00a0more than once a month.<\/p>\n<p>The goal of our collaboration was to enable NooBaa to achieve a\u00a0seamless implementation of the following:<\/p>\n<ul>\n<li>Continuous Delivery (CD) to Azure Marketplace<\/li>\n<li>CD to any of their targeted clouds (using an abstraction layer to perform the CD from Jenkins)<\/li>\n<\/ul>\n<h2>Alternatives<\/h2>\n<p>We evaluated multiple\u00a0CD offerings. \u00a0For NooBaa&#8217;s requirements, the best choice was <a href=\"http:\/\/www.spinnaker.io\/\">Spinnaker<\/a>. However, Spinnaker is better suited to deploying large environments consisting of many computers or services. In addition, Spinnaker is relatively expensive, even for small-scale deployments, which could make it less suitable for small or medium businesses (SMBs) and\/or single VMs that will\u00a0end up as offerings in the marketplace.<\/p>\n<h3>Code Alternatives<\/h3>\n<p>We found two OSS code repositories that abstract tasks that need\u00a0to be performed in the cloud:<\/p>\n<ul>\n<li><strong>pkgcloud<\/strong>: <a href=\"https:\/\/github.com\/pkgcloud\/pkgcloud\">https:\/\/github.com\/pkgcloud\/pkgcloud<\/a> (Node.js) &#8211; the only Node.js repo we found for this purpose<\/li>\n<li><strong>libcloud<\/strong>: <a href=\"https:\/\/libcloud.apache.org\/\">https:\/\/libcloud.apache.org\/<\/a> (Python)<\/li>\n<\/ul>\n<p>We ended up using pkgcloud. Since NooBaa had already written a lot of code with Node.js, pkgcloud made integrating into and importing from their existing code base a lot easier. While pkgcloud supports ASM on Azure, it does not support Azure Resource Manager (ARM).<\/p>\n<p>Therefore we collaborated with Nooba to provide ARM support for pkgcloud.<\/p>\n<h2>Solution<\/h2>\n<p>Our collaboration was divided into three parts:<\/p>\n<h3>ARM Support for pkgcloud<\/h3>\n<p>We added ARM support to pkgcloud using a <a href=\"https:\/\/github.com\/pkgcloud\/pkgcloud\/pull\/550\">pull request<\/a>. Currently, we now support compute, storage and file management on ARM.<\/p>\n<h3>Cloud CD GitHub Repo<\/h3>\n<p>We then created a new GitHub Repository: called\u00a0<strong><a href=\"https:\/\/github.com\/CatalystCode\/cloud-cd\">cloud-cd<\/a>,<\/strong>\u00a0or\u00a0<strong>Continuous Delivery Abstraction for Cloud Platforms<\/strong>. This repo abstracts the continuous delivery actions of the\u00a0<strong>Create VM<\/strong>\u00a0(using pkgcloud), Get VM (using pkgcloud), Remote Execution on Linux or Windows (using outputs from pkgcloud for running scripts on new VMs) and Destroy VM (using pkgcloud) processes.<\/p>\n<h3>Integrating cloud-cd into Jenkins<\/h3>\n<p>The integration of cloud-cd\u00a0into Jenkins enables pipeline management. We did not implement some of the pipeline actions in the scope of this solution; NooBaa plans\u00a0to do so in future.<\/p>\n<p>The implementation of cloud-cd\u00a0enables us to perform actions using simple commands such as:<\/p>\n<pre><code class=\"javascript\">cloudCD.createVM(\"azure\", azureConfig.server);\r\ncloudCD.remoteExecute(azureConfig.server, scriptPath);\r\n<\/code><\/pre>\n<h2>Jenkins Integration<\/h2>\n<p><a href=\"https:\/\/jenkins.io\/\">Jenkins<\/a> is the leading open source automation server, providing hundreds of plugins to support building, deploying and automating any project. We wanted to use NooBaa&#8217;s Jenkins server to manage the process of continuous delivery to multiple cloud marketplaces.<\/p>\n<p>Once we had\u00a0the code ready, we needed an executable Node program with parameters. For this reason, we added the following exec.js file (<a href=\"https:\/\/github.com\/CatalystCode\/cloud-cd\/blob\/master\/exec.js\">full description of the executable<\/a> available on GitHub).<\/p>\n<p><em>exec.js<\/em>:<\/p>\n<pre><code class=\"javascript\">var actions = require('.\/lib');\r\n\r\nvar action = process.argv[2];\r\nvar provider = process.argv[3];\r\n\r\nvar configString = process.argv[4];\r\nvar config = JSON.parse(configString);\r\nvar providerConfig = config[provider];\r\n\r\n\/\/ ...\r\n\r\nswitch (action) {\r\ncase 'create':\r\n\r\nvar createVMClient = new actions.CreateVMAction(providerConfig.connection);\r\ncreateVMClient.perform(providerConfig.server, function (err, server) {\r\nif (err) {\r\nreturn console.error(err);\r\n}\r\nconsole.dir(server);\r\n});\r\nbreak;\r\n\r\n\/\/ Other cases...\r\n}\r\n<\/code><\/pre>\n<p>Next, we need to call this executable from Jenkins. Let&#8217;s see how to do this call for creating a VM using the following steps:<\/p>\n<p>First, <strong>create a new job in Jenkins<\/strong>.<\/p>\n<p>Second, <strong>create 3 variables<\/strong>:<\/p>\n<ul>\n<li><code>action<\/code> = <code>create<\/code><\/li>\n<li><code>json_config<\/code> = <code>{... full json configuration ...}<\/code><\/li>\n<li><code>options<\/code> = This is optional and can be supplied for remote execute and delete<\/li>\n<\/ul>\n<p><figure id=\"attachment_2502\" aria-labelledby=\"figcaption_attachment_2502\" class=\"wp-caption alignnone\" ><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.params.png\" alt=\"Image jenkins params\" width=\"818\" height=\"722\" class=\"aligncenter size-full wp-image-11019\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.params.png 818w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.params-300x265.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.params-768x678.png 768w\" sizes=\"(max-width: 818px) 100vw, 818px\" \/><figcaption id=\"figcaption_attachment_2502\" class=\"wp-caption-text\">Jenkins Parameters<\/figcaption><\/figure><\/p>\n<p><em>config.sample.json<\/em> (available <a href=\"https:\/\/github.com\/CatalystCode\/cloud-cd\/blob\/master\/config.sample.json\">on GitHub<\/a>):<\/p>\n<pre><code class=\"json\">{\r\n\"azure-v2\": {\r\n\"connection\": {\r\n\"provider\": \"azure-v2\",\r\n\"subscriptionId\": \"1a56e85b-c5ed-4251-a1a7-995a1957dd40\",\r\n\"resourceGroup\": \"resource-group\",\r\n\"servicePrincipal\": {\r\n\"clientId\": \"c6b93eba-476c-4145-a6b9-a3c52d116331\",\r\n\"secret\": \"xasdFAsdfasdASdfasdfASdfAsdfasdfaaWrTHrfGXCVSd=\",\r\n\"domain\": \"47ad00f4-f33f-4005-b4c6-4d6f7b3b1ada\"\r\n}\r\n},\r\n\"server\": {\r\n\"name\": \"vm-server-name\",\r\n\"flavor\": \"Standard_D1\",\r\n\"username\": \"pkgcloud\",\r\n\"password\": \"Pkgcloud!!\",\r\n\"storageOSDiskName\": \"osdisk\",\r\n\"storageDataDiskNames\": [\"datadisk1\"],\r\n\"osType\": \"Linux\",\r\n\"imagePublisher\": \"Canonical\",\r\n\"imageOffer\": \"UbuntuServer\",\r\n\"imageSku\": \"16.04.0-LTS\",\r\n\"imageVersion\": \"latest\"\r\n}\r\n}\r\n}\r\n<\/code><\/pre>\n<p>Third, <strong>create an action with the following command<\/strong>:<\/p>\n<p><figure id=\"attachment_2501\" aria-labelledby=\"figcaption_attachment_2501\" class=\"wp-caption alignnone\" ><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.build_.png\" alt=\"Image jenkins build\" width=\"954\" height=\"544\" class=\"aligncenter size-full wp-image-11018\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.build_.png 954w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.build_-300x171.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2017\/04\/jenkins.build_-768x438.png 768w\" sizes=\"(max-width: 954px) 100vw, 954px\" \/><figcaption id=\"figcaption_attachment_2501\" class=\"wp-caption-text\">Jenkins &#8211; Build Configuration<\/figcaption><\/figure><\/p>\n<pre><code class=\"sh\">export NVM_DIR=\/usr\/local\/nvm\r\n. \/opt\/nvm\/nvm.sh\r\nnvm install 6.9.5\r\nnvm use 6.9.5\r\ncd $WORKSPACE\r\nnpm install\r\nnode exec.js $action azure $json_config $options\r\n<\/code><\/pre>\n<h2>Opportunities for Reuse<\/h2>\n<p>Both <strong>pkgcloud<\/strong>\u00a0and <strong>cloud-cd<\/strong>\u00a0are reusable in any scenario that requires <strong>continuous delivery of a VM image to multiple cloud marketplaces<\/strong>.<\/p>\n<p>Projects that need to <strong>perform abstracted actions on multiple cloud platforms<\/strong> \u2014 like <code>create vm<\/code>, <code>destroy storage<\/code>, etc. \u2014 can also use the knowledge from this project as a good starting point.<\/p>\n<h2>Code Repositories<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/CatalystCode\/cloud-cd\">https:\/\/github.com\/CatalystCode\/cloud-cd<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/CatalystCode\/pkgcloud\">https:\/\/github.com\/CatalystCode\/pkgcloud<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This code story describes how we collaborated with a partner to create a continuous delivery solution that runs on multiple cloud platforms using Jenkins.<\/p>\n","protected":false},"author":21371,"featured_media":11017,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[10,16],"tags":[84,119,221,276,293,378],"class_list":["post-2493","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-app-services","category-devops","tag-azure-marketplace","tag-cloud-storage","tag-jenkins-ci","tag-noobaa","tag-pkgcloud","tag-vm-image"],"acf":[],"blog_post_summary":"<p>This code story describes how we collaborated with a partner to create a continuous delivery solution that runs on multiple cloud platforms using Jenkins.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21371"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2493"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2493\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/11017"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}