Part 5 – Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process

Developer Support

John Folberth and Joe Fitzgerald share sample guidance for developing and deploying an Azure Data Factory into multiple environments.

Introduction 

Part 1

1. Architecture and Scenario
2. Creating resources in Azure
3. Create Azure Storage Containers
4. Create Azure Key Vaults
5. Create Azure Data Factory: With Key Vault Access

Part 2 

1. Configure Azure Data Factory Source Control
2. Construct Azure Data Factory Data Pipeline
3. Publishing Concept for Azure Data Factory
4. Configure Deployed Azure Resources.

Part 3

1. The YAML Pipeline Structure
2. The Publish Process
3. ARM Template Parameterization
4. ADF ARM Template Deployment

Part 4

1. How to use Azure DevOps Pipeline Templates

This section will build on that and include how to build/deploy larger Data Factories which leverage Azure Resource Manager Linked Templates for deployment. 

When Linked Templates Are Required 

Typically, users will not desire to use linked templates; however, there are explicit limitations of a single ARM template which may require one to deploy their ARM template via Linked Templates. Specifically, when the ARM template.json file is > 4MB is one such instance.

Regardless of the Data Factory size the Data Factory build process will generate the necessary files and file structure to accommodate both a single template deployment and a linked ARM template deployment. The linked templates are only required if your main ARM template file is greater than >4MB. (i.e. if you have a larger Data Factory). 

What is a Linked Template 

A linked template is a process by which the main deployment template will call additional files (‘linked’). These files will need to stored in an Azure Storage Account. This will break up the single 4 MB file limitation of an ARM template as it will call multiple files whose size is < 4MB. Now to be clear you will still be limited by limitation of an ARM template. As a reminder those are: 

• 256 parameters 
• 256 variables 
• 800 resources (including copy count) 
• 64 output values 

• 10 unique locations per subscription/tenant/management group scope 
• 24,576 characters in a template expression 

Check out full post (Part 5) and the series in the Healthcare and Life Sciences Tech Community here.

Developer Support App Dev Customer Success Account Manager, Microsoft Developer Support

Follow