{"id":328,"date":"2020-08-14T08:06:38","date_gmt":"2020-08-14T15:06:38","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/azure-sql\/?p=328"},"modified":"2020-08-25T02:41:21","modified_gmt":"2020-08-25T09:41:21","slug":"developing-in-the-cloud-with-sql-server-big-data-clusters-develop-and-deploy-apps","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/azure-sql\/developing-in-the-cloud-with-sql-server-big-data-clusters-develop-and-deploy-apps\/","title":{"rendered":"Developing in the cloud with SQL Server Big Data Clusters: Develop and Deploy Apps"},"content":{"rendered":"<p>Kubernetes is designed to support an open platform to support a diverse variety of containerized workloads including both stateless and stateful and data-processing workloads. Technically any application can be containerized, it can be up and running in Kubernetes. As a cloud-native platform, Kubernetes is available to deploy cloud-native applications in any language of your choice. Working with BDC, you get more benefits on the top of Kubernetes.<\/p>\n<p>This article will walk you through how to develop and deploy apps to Big Data Clusters ( BDC ).<\/p>\n<h2>BDC Native versus Kubernetes Native<\/h2>\n<p>Applications deployed on BDC are not only to benefit from numerous advantages, such as the computational power of the cluster but can also access massive data that is available on the cluster. This can be dramatically beneficial to increase the performance of the applications since it sits in the same area where the data lives.\u00a0\u00a0The natively supported application runtimes on SQL Server Big Data Clusters are as the following:<\/p>\n<ul>\n<li><strong>Python<\/strong> &#8211; One of the most popular general programming languages for various personas for instance Data Engineers, Data Scientists, and DevOps engineers. It supports numerous scenarios such as data wrangling, automation, prototyping, to some extent. It also increasingly used to program enterprise-grade applications working in conjunction with web development frameworks such as Flask and Django to address different business requirements.<\/li>\n<li><strong>R\u00a0<\/strong>\u2013 Another popular programming language for Data Engineers and Data Scientists. Compared to Python, R is a programming language with more a specific focus on statistical computing and graphics.<\/li>\n<li><strong>SQL Server Integration Services (SSIS)<\/strong> \u00a0&#8211;\u00a0\u00a0high-performance data integration solutions\u00a0for building and debugging\u00a0ETL\u00a0packages, it uses\u00a0Data Transformation Services Package File Format (DTSX)\u00a0which\u00a0is an XML-based file format that stores the instructions for the processing of\u00a0migrating data between databases\u00a0and\u00a0the integration of external data sources.<\/li>\n<li><strong>MLeap<\/strong>\u00a0&#8211;\u00a0is a common serialization format\u00a0and\u00a0provides everything needed to execute and serialize\u00a0SparkML\u00a0pipelines\u00a0and\u00a0<a href=\"https:\/\/github.com\/combust\/mleap\">others<\/a>\u00a0that can\u00a0then\u00a0be\u00a0loaded\u00a0at runtime\u00a0to\u00a0process\u00a0ML scoring\u00a0tasks\u00a0in\u00a0near\u00a0real-time\u00a0and close to\u00a0the\u00a0data.<\/li>\n<\/ul>\n<p>Find those of samples of entre level application that can be deployed in BDC on <a href=\"https:\/\/github.com\/Microsoft\/sql-server-samples\/tree\/master\/samples\/features\/sql-big-data-cluster\/app-deploy\">Github<\/a>.<\/p>\n<h2>Develop and Deploy\u00a0Application on BDC<\/h2>\n<p>One of the coolest\u00a0things\u00a0while working with BDC is we can start application development by\u00a0creating an app\u00a0skeleton\u00a0by using\u00a0<strong>azdata\u00a0app\u00a0init<\/strong> command which provides a scaffold with the relevant artifacts that are required for deploying an app. For instance, to start a python application, we can use the following command:<\/p>\n<pre class=\"prettyprint\">azdata\u00a0app\u00a0init\u00a0--name hello --version v1 --template python<\/pre>\n<p>It creates a\u00a0folder\u00a0structure\u00a0as the following:<\/p>\n<ul>\n<li><strong>hello.py<\/strong> where your python code lives. It can be another particular type of language kernel. It supports Python and R by far.<\/li>\n<li><strong>run-spec.yaml<\/strong> contains everything app consumption definition given to the VS Code extension client. To know more about how to work with VS Code App Deploy extension, check <a href=\"https:\/\/docs.microsoft.com\/en-us\/sql\/big-data-cluster\/app-deployment-extension?view=sql-server-ver15\">here<\/a>.<\/li>\n<li><strong>spec.yaml<\/strong> contains everything the controller needs to know to successfully deploy the application in BDC.<\/li>\n<\/ul>\n<p>If you open this folder with VS Code or any other IDE of your choice. You\u2019d see a screenshot as the following screenshot:<\/p>\n<p><img decoding=\"async\" class=\"wp-image-332 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/Spec.png\" alt=\"Image Spec\" width=\"908\" height=\"578\" \/><\/p>\n<p>Then we\u2019ll deploy the sample application by using <strong>azdata\u00a0app create<\/strong>\u00a0command, and then specify the\u00a0<strong>folder\u00a0path<\/strong> to your\u00a0<strong>spec.yaml<\/strong>\u00a0file. In case you\u2019re on Linux OS,\u00a0you can use\u00a0<strong>pwd\u00a0<\/strong>command to get your current path :<\/p>\n<pre class=\"prettyprint\">azdata\u00a0app create \u2013-spec\u00a0\/path\/to\/your\/spec.yaml<\/pre>\n<p>If you\u2019re already in the folder where you have your spec.yaml file,\u00a0 use the following command:<\/p>\n<pre class=\"prettyprint\">azdata app create --spec .<\/pre>\n<p>After executing this command, you can expect the\u00a0following output which means you have deployed an application successfully.<\/p>\n<p><img decoding=\"async\" class=\"wp-image-334 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-Created.png\" alt=\"Image App Created\" width=\"407\" height=\"24\" srcset=\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-Created.png 1218w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-Created-300x17.png 300w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-Created-1024x60.png 1024w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-Created-768x45.png 768w\" sizes=\"(max-width: 407px) 100vw, 407px\" \/><\/p>\n<p>Let\u2019s\u00a0check\u00a0all applications deployed in BDC\u00a0using\u00a0\u00a0azdata\u00a0app list\u00a0command, notice that\u00a0<strong>sqlbigdata\u00a0<\/strong>is the <strong>name of your BDC cluster<\/strong> which is also the name of the namespace at this time:<\/p>\n<pre class=\"prettyprint\">azdata\u00a0app list<\/pre>\n<p>If you have a similar output as follows with <strong>Ready<\/strong> state which means you have successfully deployed the application in BDC and it&#8217;s ready to be consumed via a respective endpoint.<\/p>\n<p><img decoding=\"async\" class=\"wp-image-333 aligncenter\" src=\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List.png\" alt=\"Image App List\" width=\"880\" height=\"79\" srcset=\"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List.png 2048w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List-300x27.png 300w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List-1024x92.png 1024w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List-768x69.png 768w, https:\/\/devblogs.microsoft.com\/azure-sql\/wp-content\/uploads\/sites\/56\/2020\/08\/App-List-1536x137.png 1536w\" sizes=\"(max-width: 880px) 100vw, 880px\" \/><\/p>\n<h2>Next Steps<\/h2>\n<p>Next article we\u2019ll walk through how to develop and deploy Apps to SQL Server Big Data Clusters (BDC), please check: Working with big data analytics solution in Kubernetes Part 3 &#8211; <a href=\"https:\/\/devblogs.microsoft.com\/azure-sql\/developing-in-the-cloud-with-sql-server-big-data-clusters-run-and-monitor-apps\/\">Run and Monitor Apps in SQL Server Big Data Clusters (BDC)<\/a>. Let\u2019s stay tuned!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Kubernetes is designed to support an open platform to support a diverse variety of containerized workloads including both stateless and stateful and data-processing workloads. Technically any application can be containerized, it can be up and running in Kubernetes. As a cloud-native platform, Kubernetes is available to deploy cloud-native applications in any language of your choice. [&hellip;]<\/p>\n","protected":false},"author":32492,"featured_media":346,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,283,92],"tags":[279,284,278,277],"class_list":["post-328","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sql","category-bdc","category-devops","tag-aks","tag-bdc","tag-kubernetes","tag-sql-server-big-data-clusters"],"acf":[],"blog_post_summary":"<p>Kubernetes is designed to support an open platform to support a diverse variety of containerized workloads including both stateless and stateful and data-processing workloads. Technically any application can be containerized, it can be up and running in Kubernetes. As a cloud-native platform, Kubernetes is available to deploy cloud-native applications in any language of your choice. [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts\/328","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/users\/32492"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/comments?post=328"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/posts\/328\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/media\/346"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/media?parent=328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/categories?post=328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sql\/wp-json\/wp\/v2\/tags?post=328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}