Part 5 – Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process
- Architecture and Scenario
- Creating resources in Azure
- Create Azure Storage Containers
- Create Azure Key Vaults
- Create Azure Data Factory: With Key Vault Access
- Configure Azure Data Factory Source Control
- Construct Azure Data Factory Data Pipeline
- Publishing Concept for Azure Data Factory
- Configure Deployed Azure Resources.
- The YAML Pipeline Structure
- The Publish Process
- ARM Template Parameterization
- ADF ARM Template Deployment
- How to use Azure DevOps Pipeline Templates
This section will build on that and include how to build/deploy larger Data Factories which leverage Azure Resource Manager Linked Templates for deployment.
When Linked Templates Are Required
Typically, users will not desire to use linked templates; however, there are explicit limitations of a single ARM template which may require one to deploy their ARM template via Linked Templates. Specifically, when the ARM template.json file is > 4MB is one such instance.
Regardless of the Data Factory size the Data Factory build process will generate the necessary files and file structure to accommodate both a single template deployment and a linked ARM template deployment. The linked templates are only required if your main ARM template file is greater than >4MB. (i.e. if you have a larger Data Factory).
What is a Linked Template
A linked template is a process by which the main deployment template will call additional files (‘linked’). These files will need to stored in an Azure Storage Account. This will break up the single 4 MB file limitation of an ARM template as it will call multiple files whose size is < 4MB. Now to be clear you will still be limited by limitation of an ARM template. As a reminder those are:
- 256 parameters
- 256 variables
- 800 resources (including copy count)
- 64 output values
- 10 unique locations per subscription/tenant/management group scope
- 24,576 characters in a template expression
Check out full post (Part 5) and the series in the Healthcare and Life Sciences Tech Community here.