{"id":41517,"date":"2023-10-31T00:13:00","date_gmt":"2023-10-31T07:13:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/premier-developer\/?p=41517"},"modified":"2023-10-30T16:21:47","modified_gmt":"2023-10-30T23:21:47","slug":"part-5-unlock-the-power-of-azure-data-factory-a-guide-to-boosting-your-data-ingestion-process","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/part-5-unlock-the-power-of-azure-data-factory-a-guide-to-boosting-your-data-ingestion-process\/","title":{"rendered":"Part 5 – Unlock the Power of Azure Data Factory: A Guide to Boosting Your Data Ingestion Process"},"content":{"rendered":"

John Folberth<\/a>\u00a0and\u00a0Joe Fitzgerald<\/a>\u00a0share sample guidance for developing and deploying an Azure Data Factory into multiple environments.<\/p>\n

\n

Introduction<\/span>\u00a0<\/span><\/h2>\n

Part 1<\/a><\/p>\n

    \n
  1. Architecture and Scenario<\/li>\n
  2. Creating resources in Azure<\/li>\n
  3. Create Azure Storage Containers<\/li>\n
  4. Create Azure Key Vaults<\/li>\n
  5. Create Azure Data Factory: With Key Vault Access<\/li>\n<\/ol>\n

    Part 2\u00a0<\/a><\/p>\n

      \n
    1. Configure Azure Data Factory Source Control<\/li>\n
    2. Construct Azure Data Factory Data Pipeline<\/li>\n
    3. Publishing Concept for Azure Data Factory<\/li>\n
    4. Configure Deployed Azure Resources.<\/li>\n<\/ol>\n

      Part 3<\/a><\/p>\n

        \n
      1. The YAML Pipeline Structure<\/li>\n
      2. The Publish Process<\/li>\n
      3. ARM Template Parameterization<\/li>\n
      4. ADF ARM Template Deployment<\/li>\n<\/ol>\n

        Part 4<\/a><\/p>\n

          \n
        1. How to use Azure DevOps Pipeline Templates<\/li>\n<\/ol>\n

          This section will build on that and include how to build\/deploy larger Data Factories which leverage\u00a0<\/span>Azure Resource Manager Linked Templates<\/span><\/a>\u00a0for deployment.<\/span>\u00a0<\/span><\/p>\n

           <\/p>\n

          When Linked Templates Are Required<\/span>\u00a0<\/span><\/h2>\n

          Typically, users will not desire to use linked templates; however, there are explicit limitations of a single ARM template which may require one to deploy their ARM template via Linked Templates. Specifically, when the ARM template.json file is > 4MB is one such instance.<\/span><\/p>\n

           <\/p>\n

          Regardless of the Data Factory size the Data Factory build process will generate the necessary files and file structure to accommodate both a single template deployment and a linked ARM template deployment. The linked templates are only required if your main ARM template file is greater than >4MB. (i.e. if you have a larger Data Factory).<\/span>\u00a0<\/span><\/p>\n

           <\/p>\n

          What is a Linked Template<\/span>\u00a0<\/span><\/h2>\n

          A linked template is a process by which the main deployment template will call additional files (\u2018linked\u2019). These files will need to stored in an Azure Storage Account. This will break up the single 4 MB file limitation of an ARM template as it will call multiple files whose size is < 4MB. Now to be clear you will still be limited by limitation of an ARM template.\u00a0<\/span>As a reminder those are<\/span><\/a>:<\/span>\u00a0<\/span><\/p>\n