{"id":41464,"date":"2023-09-15T01:00:29","date_gmt":"2023-09-15T08:00:29","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/premier-developer\/?p=41464"},"modified":"2023-09-13T11:12:43","modified_gmt":"2023-09-13T18:12:43","slug":"part-4-unlock-the-power-of-azure-data-factory-a-guide-to-boosting-your-data-ingestion-process","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/part-4-unlock-the-power-of-azure-data-factory-a-guide-to-boosting-your-data-ingestion-process\/","title":{"rendered":"Part 4 &#8211; Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process"},"content":{"rendered":"<p><a href=\"https:\/\/www.linkedin.com\/in\/john-folberth\/\" target=\"_blank\" rel=\"noopener\">John Folberth<\/a>\u00a0and\u00a0<a href=\"https:\/\/www.linkedin.com\/in\/joseph-t-fitzgerald\/\" target=\"_blank\" rel=\"noopener\">Joe Fitzgerald<\/a>\u00a0share sample guidance for developing and deploying an Azure Data Factory into multiple environments.<\/p>\n<hr \/>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2023\/09\/azpipes_fi.jpg\"><img decoding=\"async\" class=\"alignnone size-full wp-image-41465\" src=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2023\/09\/azpipes_fi.jpg\" alt=\"Image azpipes fi\" width=\"400\" height=\"113\" srcset=\"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2023\/09\/azpipes_fi.jpg 400w, https:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2023\/09\/azpipes_fi-300x85.jpg 300w\" sizes=\"(max-width: 400px) 100vw, 400px\" \/><\/a><\/p>\n<h2 id=\"toc-hId--1241980115\" aria-level=\"2\"><span data-contrast=\"none\">Background<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">This post is the next post in the series\u00a0<a href=\"https:\/\/techcommunity.microsoft.com\/t5\/healthcare-and-life-sciences\/bg-p\/HealthcareAndLifeSciencesBlog\/label-name\/CICD%20ADF\" target=\"_self\" rel=\"noopener\">Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process<\/a>. This also happens to overlap and is included in the\u00a0<\/span><a href=\"https:\/\/techcommunity.microsoft.com\/t5\/healthcare-and-life-sciences\/bg-p\/HealthcareAndLifeSciencesBlog\/label-name\/YAML%20Pipeline%20Series\" target=\"_blank\" rel=\"noopener\"><span data-contrast=\"none\">series on YAML Pipelines<\/span><\/a><span data-contrast=\"auto\">.\u00a0 All code snippets and final templates can be found out on my GitHub\u00a0<\/span><a href=\"https:\/\/github.com\/JFolberth\/TheYAMLPipelineOne\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-contrast=\"none\">TheYAMLPipelineOne.<\/span><\/a><span data-contrast=\"auto\">\u00a0For the actual data factory, we will leverage my\u00a0<\/span><a href=\"https:\/\/github.com\/JFolberth\/adf_pipelines_yaml_ci_cd\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-contrast=\"none\">adf_pipelines_yaml_ci_cd<\/span><\/a><span data-contrast=\"auto\">\u00a0repository.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p>&nbsp;<\/p>\n<h2 id=\"toc-hId-1245532718\" aria-level=\"2\"><span data-contrast=\"none\">Introduction<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">After reading parts 1-3 on\u00a0<a href=\"https:\/\/techcommunity.microsoft.com\/t5\/healthcare-and-life-sciences\/bg-p\/HealthcareAndLifeSciencesBlog\/label-name\/CICD%20ADF\" target=\"_self\" rel=\"noopener\">Unlock the Power of Azure Data Factory<\/a>\u00a0one may be left with the next steps of how to take what was provided and convert it to an enterprise scale. Terminology and expectations are key so let\u2019s outline what we would like to see from an enterprise-scale deployment:<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li><span data-contrast=\"auto\">Write once reuse across projects.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"auto\">Individual components can be reused.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"auto\">Limited manual intervention.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"auto\">Easily updated.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<li><span data-contrast=\"auto\">Centralized definition.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">Depending on where your organization is at in your pipeline and DevOps maturity this may sound daunting. Have no fear as we will walk you through how to achieve this with YAML templates for Azure Data Factory. At the end of this piece, you should be well equipped to create a working pipeline for Azure Data Factory in a manner of minutes.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p>&nbsp;<\/p>\n<h2 id=\"toc-hId--561921745\" aria-level=\"2\"><span data-contrast=\"none\">Set Up<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">To assist in the goals outlined above for enterprise scale deployments, I recommend having a separate repository for your YAML templates that resides outside of your Data Factory. This will help check off the boxes on a centralized definition, write once and reuse across projects, easily updated, and individual components that can be reused.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0For more context on this check out my post on\u00a0<a href=\"https:\/\/techcommunity.microsoft.com\/t5\/healthcare-and-life-sciences\/azure-devops-pipelines-practices-for-scaling-templates\/ba-p\/3855524\" target=\"_self\" rel=\"noopener\">Azure DevOps Pipelines: Practices for Scaling Templates<\/a>.<\/span><\/p>\n<p>&nbsp;<\/p>\n<p><span data-contrast=\"auto\">Our individual Data Factories will each have a dedicated CI\/CD pipeline which will reference the separate repository we are putting the YAML templates in. This can be\u00a0<\/span><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/devops\/pipelines\/repos\/multi-repo-checkout?view=azure-devops#specify-multiple-repositories\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-contrast=\"none\">achieved natively in Azure DevOps<\/span><\/a><span data-contrast=\"auto\">. Furthermore, it is not unheard of for larger scale organizations to have a \u201cDevOps Team\u201d or a team responsible for pipeline deployments. If this is the case in your organization, you can think of this other team as \u201cowning\u201d the centralized repository.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p>Check out full post and <a href=\"https:\/\/techcommunity.microsoft.com\/t5\/healthcare-and-life-sciences\/part-4-unlock-the-power-of-azure-data-factory-a-guide-to\/ba-p\/3872590\" target=\"_blank\" rel=\"noopener\">the series in the Healthcare and Life Sciences Tech Community here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>John Folberth\u00a0and\u00a0Joe Fitzgerald\u00a0share sample guidance for developing and deploying an Azure Data Factory into multiple environments. Background\u00a0 This post is the next post in the series\u00a0Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process. This also happens to overlap and is included in the\u00a0series on YAML Pipelines.\u00a0 All code [&hellip;]<\/p>\n","protected":false},"author":582,"featured_media":41465,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25,10643],"tags":[154,2571],"class_list":["post-41464","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-azure-data-factory","tag-azure-data-factory","tag-azure-devops"],"acf":[],"blog_post_summary":"<p>John Folberth\u00a0and\u00a0Joe Fitzgerald\u00a0share sample guidance for developing and deploying an Azure Data Factory into multiple environments. Background\u00a0 This post is the next post in the series\u00a0Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process. This also happens to overlap and is included in the\u00a0series on YAML Pipelines.\u00a0 All code [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/41464","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=41464"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/41464\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/41465"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=41464"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=41464"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=41464"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}