Part 5 – Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process

Developer Support

John Folberth and Joe Fitzgerald share sample guidance for developing and deploying an Azure Data Factory into multiple environments.


Part 1

  1. Architecture and Scenario
  2. Creating resources in Azure
  3. Create Azure Storage Containers
  4. Create Azure Key Vaults
  5. Create Azure Data Factory: With Key Vault Access

Part 2 

  1. Configure Azure Data Factory Source Control
  2. Construct Azure Data Factory Data Pipeline
  3. Publishing Concept for Azure Data Factory
  4. Configure Deployed Azure Resources.

Part 3

  1. The YAML Pipeline Structure
  2. The Publish Process
  3. ARM Template Parameterization
  4. ADF ARM Template Deployment

Part 4

  1. How to use Azure DevOps Pipeline Templates

This section will build on that and include how to build/deploy larger Data Factories which leverage Azure Resource Manager Linked Templates for deployment. 


When Linked Templates Are Required 

Typically, users will not desire to use linked templates; however, there are explicit limitations of a single ARM template which may require one to deploy their ARM template via Linked Templates. Specifically, when the ARM template.json file is > 4MB is one such instance.


Regardless of the Data Factory size the Data Factory build process will generate the necessary files and file structure to accommodate both a single template deployment and a linked ARM template deployment. The linked templates are only required if your main ARM template file is greater than >4MB. (i.e. if you have a larger Data Factory). 


What is a Linked Template 

A linked template is a process by which the main deployment template will call additional files (‘linked’). These files will need to stored in an Azure Storage Account. This will break up the single 4 MB file limitation of an ARM template as it will call multiple files whose size is < 4MB. Now to be clear you will still be limited by limitation of an ARM template. As a reminder those are: 

  • 256 parameters 
  • 256 variables 
  • 800 resources (including copy count) 
  • 64 output values 
  • 10 unique locations per subscription/tenant/management group scope 
  • 24,576 characters in a template expression 

Check out full post (Part 5) and the series in the Healthcare and Life Sciences Tech Community here.


Discussion is closed.

Feedback usabilla icon