{"id":5170,"date":"2022-12-13T07:28:44","date_gmt":"2022-12-13T15:28:44","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=5170"},"modified":"2022-12-02T06:38:41","modified_gmt":"2022-12-02T14:38:41","slug":"under-the-hood-of-the-new-azure-functions-extension-for-azure-cosmos-db","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/under-the-hood-of-the-new-azure-functions-extension-for-azure-cosmos-db\/","title":{"rendered":"Under the hood of the new Azure Functions extension for Azure Cosmos DB"},"content":{"rendered":"<p>The <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.WebJobs.Extensions.CosmosDB\" target=\"_blank\" rel=\"noopener\">Azure Cosmos DB Azure Functions extension version 4<\/a>\u00a0is now GA, and it packs a load of improvements and new features. The goal of this post is to go deep into the technical details about the extension changes and guide you to take advantage of them.<\/p>\n<h4>Source code<\/h4>\n<p>As a starting point, everything I&#8217;ll be talking about in the article is available on the <a href=\"https:\/\/github.com\/Azure\/azure-webjobs-sdk-extensions\/tree\/dev\/src\/WebJobs.Extensions.CosmosDB\" target=\"_blank\" rel=\"noopener\">GitHub repository<\/a> where the extension source code is developed. That is the best place to post and share your <strong>feedback<\/strong>. The following sections will point to the source code as reference for those curious on how each feature works.<\/p>\n<h4>Managed identity authentication<\/h4>\n<p>Azure resources support <a href=\"https:\/\/learn.microsoft.com\/azure\/active-directory\/managed-identities-azure-resources\/overview\" target=\"_blank\" rel=\"noopener\">managed identities<\/a>, and Azure Functions is not the exception. Our Azure Function can have a <a href=\"https:\/\/learn.microsoft.com\/azure\/app-service\/overview-managed-identity\" target=\"_blank\" rel=\"noopener\">system assigned managed identity<\/a> that represents it in the Azure Active Directory tenant for our subscription.<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad1.png\">\n<img decoding=\"async\" class=\"aligncenter size-full wp-image-5174\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad1.png\" alt=\"Shows how to enable managed identities for our Function on the Azure Portal UI \" width=\"717\" height=\"357\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad1.png 717w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad1-300x149.png 300w\" sizes=\"(max-width: 717px) 100vw, 717px\" \/><\/a><\/p>\n<p>If we combine this with <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/how-to-setup-rbac\" target=\"_blank\" rel=\"noopener\">Azure Cosmos DB&#8217;s role-based access control<\/a>, we can assign data access permissions to the Function&#8217;s identity directly. Because the new extension is using the latest Azure Cosmos DB .NET SDK, which supports <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/microsoft.azure.cosmos.cosmosclient.-ctor?view=azure-dotnet#microsoft-azure-cosmos-cosmosclient-ctor(system-string-azure-core-tokencredential-microsoft-azure-cosmos-cosmosclientoptions)\" target=\"_blank\" rel=\"noopener\">authenticating using TokenCredentials<\/a>, we can make this work!<\/p>\n<p>The new extension changes the &#8220;ConnectionStringSetting&#8221; property for a <strong>&#8220;Connection&#8221;<\/strong> one, the big difference (besides the rename) is that the value can point to configuration that specifies <a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-reference?tabs=cosmos#connections\">different authentication methods<\/a>. The extension code will obtain the value of the configuration from the environment and <a href=\"https:\/\/github.com\/Azure\/azure-webjobs-sdk-extensions\/blob\/dev\/src\/WebJobs.Extensions.CosmosDB\/Config\/DefaultCosmosDBServiceFactory.cs#L38-L66\">parse it<\/a>. If the configuration is a Connection String, it will use that value to create the CosmosClient. If the configuration is a managed identity, it will call the <a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/blob\/736b0f5649db1d82bbf2f66568cf261e0270c182\/sdk\/extensions\/Microsoft.Extensions.Azure\/src\/Internal\/AzureComponentFactoryImpl.cs#L25\">AzureComponentFactory<\/a> to resolve the configuration into a TokenCredential, and create the CosmosClient with that.<\/p>\n<p>The steps to make it work are:<\/p>\n<ul>\n<li><strong>Enable the system assigned identity<\/strong> on the Azure Function<\/li>\n<li>Assign the desired permissions by <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/how-to-setup-rbac#role-assignments\">giving this identity a role<\/a><\/li>\n<li>Add the <strong>Azure Functions Configuration<\/strong> for the name of the &#8220;Connection&#8221; property in the Function that contains the endpoint (<code>&lt;ConnectionPropertyValue&gt;__accountEndpoint<\/code>) and credential (<code>&lt;ConnectionPropertyValue&gt;__credential<\/code>). Using the special value &#8220;managedidentity&#8221; as credential tells the Azure Functions runtime to use the system assigned identity. The below screenshot shows the example of a Function with a &#8220;Connection&#8221; property with value &#8220;MyConnection&#8221; and connecting to an Azure Cosmos DB account with the endpoint &#8220;https:\/\/mycosmosdbaccount.documents.azure.com:443\/&#8221;:<\/li>\n<\/ul>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-5175\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2.png\" alt=\"Image showing the configuration on Azure Functions with endpoint and credential\" width=\"1452\" height=\"311\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2.png 1452w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2-300x64.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2-1024x219.png 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2022\/10\/aad2-768x164.png 768w\" sizes=\"(max-width: 1452px) 100vw, 1452px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<h4>Improved logging<\/h4>\n<p>The Azure Functions trigger for Azure Cosmos DB (AKA CosmosDBTrigger) leverages the <a href=\"https:\/\/docs.microsoft.com\/azure\/cosmos-db\/sql\/change-feed-processor\">Change Feed Processor<\/a> to dynamically distribute work across Function instances. This component handles work mainly as a background process. As any background process, it can run into failure conditions or problems. Because these failures do not happen on the context where the Function code runs, they are not easily viewable.<\/p>\n<p>The new extension is using a newer Change Feed Processor, and we are <a href=\"https:\/\/github.com\/Azure\/azure-webjobs-sdk-extensions\/blob\/dev\/src\/WebJobs.Extensions.CosmosDB\/Trigger\/CosmosDBTriggerHealthMonitor.cs\">wiring Azure Function logs<\/a> to the different <a href=\"https:\/\/docs.microsoft.com\/azure\/cosmos-db\/sql\/change-feed-processor?tabs=dotnet#life-cycle-notifications\">life-cycle notifications<\/a>. You can now receive logs that help you to:<\/p>\n<ul>\n<li>Identify processing failures in your Function code<\/li>\n<li>Identify connectivity issues affecting the CosmosDBTrigger operations<\/li>\n<li>Know when an instance started or stopped processing changes<\/li>\n<li>Know when the CosmosDBTrigger has delivered data to your Function code<\/li>\n<li>Obtain detailed network diagnostics to troubleshoot latency<\/li>\n<\/ul>\n<p>These logs are critical to troubleshoot different scenarios. Remember to <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/how-to-configure-cosmos-db-trigger#enabling-trigger-specific-logs\">enable the extension specific logs<\/a> to gain access to these insights.<\/p>\n<h4>Custom serialization<\/h4>\n<p>The previous extension version used the Azure Cosmos DB SDK V2 (depending on <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.DocumentDB.Core\" target=\"_blank\" rel=\"noopener\">Microsoft.Azure.DocumentDB.Core<\/a> package). This SDK used Newtonsoft.Json as serialization engine. The new extension, which uses the Azure Cosmos DB SDK V3 (<a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.Cosmos\" target=\"_blank\" rel=\"noopener\">Microsoft.Azure.Cosmos<\/a> package) also uses Newtonsoft.Json as serialization engine but it gives the user the freedom of <strong>replacing the engine with any other<\/strong>. The V3 SDK allows users to provide a custom implementation of the <a href=\"https:\/\/learn.microsoft.com\/dotnet\/api\/microsoft.azure.cosmos.cosmosserializer\" target=\"_blank\" rel=\"noopener\">CosmosSerializer<\/a> class with any serialization technology they want, for example, <a href=\"https:\/\/github.com\/Azure\/azure-cosmos-dotnet-v3\/blob\/master\/Microsoft.Azure.Cosmos.Samples\/Usage\/SystemTextJson\/CosmosSystemTextJsonSerializer.cs\" target=\"_blank\" rel=\"noopener\">System.Text.Json<\/a>.<\/p>\n<p>To achieve this, you just need to declare, on your <code>FunctionsStartup<\/code> class, an implementation of the <code>ICosmosDBSerializerFactory<\/code> interface that returns instances of your custom serializer:<\/p>\n<p>https:\/\/gist.github.com\/ealsur\/d42abe66f04c092285452e7b4d5c5145<\/p>\n<p>When the internal <code>CosmosClient<\/code> instances are created, they will <a href=\"https:\/\/github.com\/Azure\/azure-webjobs-sdk-extensions\/blob\/dev\/src\/WebJobs.Extensions.CosmosDB\/Config\/CosmosDBExtensionConfigProvider.cs#L113\" target=\"_blank\" rel=\"noopener\">use your factory to customize the Serializer<\/a>. Any POCO type used in the Trigger, Input or Output bindings will now be deserialized with your custom serializer.<\/p>\n<p>Which leads us to another change: You are <strong>no longer limited<\/strong> to using the <code>Document<\/code> type for your Trigger definitions, you can now use the POCO type of your choice that matches your serialization engine, for example:<\/p>\n<p>https:\/\/gist.github.com\/ealsur\/320fcda298d91c59507437a8e9649fd5<\/p>\n<h4>User-agent stamping<\/h4>\n<p>Another common scenario is having multiple Functions working with the same Azure Cosmos DB account. There are <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/monitor?tabs=azure-diagnostics\" target=\"_blank\" rel=\"noopener\">several monitoring options<\/a> for your account. With some of them, you can identify which application is performing those operations through the User Agent. By using the new extension version, you can now add a <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/how-to-configure-cosmos-db-trigger#customizing-the-user-agent\" target=\"_blank\" rel=\"noopener\">custom identifier<\/a> to the requests made by each Function and the identifier will appear in the User Agent in the monitoring options!<\/p>\n<p>Just go to your <a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-host-json\" target=\"_blank\" rel=\"noopener\">host.json<\/a> in your Function project and add the <code>userAgentSuffix<\/code> on the <code>cosmosDB<\/code> section:<\/p>\n<p>https:\/\/gist.github.com\/ealsur\/a4296b0cfba29a615b5ae72e9d1da8d3<\/p>\n<h4>Retry policies<\/h4>\n<p>The CosmosDBTrigger <strong>does not retry<\/strong> on failed executions <strong>by default<\/strong>, this design aligns with other event-based triggers (like Event Hub). Using a &#8220;poison queue&#8221; is often a general recommendation. With this approach, all the documents that failed processing are sent to the queue and retried or investigated afterwards, but there is another alternative. Azure Functions <strong>added support<\/strong> for <a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-bindings-error-pages?tabs=fixed-delay%2Cin-process&amp;pivots=programming-language-csharp#retry-policies\" target=\"_blank\" rel=\"noopener\">Retry Policies<\/a>, which can be defined at the Function level using decorators that currently support fixed delay and exponential delay in-between retries. The new extension version <a href=\"https:\/\/github.com\/Azure\/azure-webjobs-sdk-extensions\/pull\/798\" target=\"_blank\" rel=\"noopener\">added support for these Retry Policies<\/a>.<\/p>\n<p>Keep in mind that this retry won&#8217;t only retry on individual failed documents, but the <strong>entire batch<\/strong>. If your Trigger delivered 100 events, and your code threw an unhandled exception on number 50, the retry will execute the Function again with all 100 events as input. So, be aware of <strong>potential duplicate processing<\/strong> when writing your Function.<\/p>\n<p>&nbsp;<\/p>\n<h4>Using the CosmosClient directly<\/h4>\n<p>Because the extension let&#8217;s you access the <code>CosmosClient<\/code> directly, you can also <strong>take advantage of all the available new SDK features<\/strong>:<\/p>\n<p>https:\/\/gist.github.com\/ealsur\/913387a2414efaf4a5670c26dc55baea<\/p>\n<p>You can leverage <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/transactional-batch\" target=\"_blank\" rel=\"noopener\">TransactionalBatch<\/a>, <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/partial-document-update-getting-started\" target=\"_blank\" rel=\"noopener\">Patch<\/a>, <a href=\"https:\/\/github.com\/ealsur\/ondotnet-cosmosdb\/tree\/master\/src\/episode1\/streams#streams\" target=\"_blank\" rel=\"noopener\">Stream APIs<\/a>, even consuming the <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/change-feed-pull-model\" target=\"_blank\" rel=\"noopener\">Change Feed as a pull model<\/a>, among other things.<\/p>\n<h4>How do I migrate?<\/h4>\n<p>Given this is a major version change of a package, it might contain some breaking changes in your code. The best migration path is:<\/p>\n<ol>\n<li>Update the <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.WebJobs.Extensions.CosmosDB\" target=\"_blank\" rel=\"noopener\">extension package<\/a> (or bundle) to the latest available version 4.X<\/li>\n<li>Apply attribute renames:\n<ol>\n<li>Anything named Collection is now Container<\/li>\n<li>UseMultipleWriteLocations is no longer needed, it is automatically detected<\/li>\n<li>UseDefaultJsonSerialization is no longer needed, you can fully customize the serialization if you want<\/li>\n<li>ConnectionStringSetting is now Connection<\/li>\n<\/ol>\n<\/li>\n<li>Use the <a href=\"https:\/\/docs.microsoft.com\/azure\/cosmos-db\/sql\/migrate-dotnet-v3\" target=\"_blank\" rel=\"noopener\">Azure Cosmos DB .NET SDK migration guide<\/a> for moving from V2 .NET SDK types to V3 .NET SDK types.<\/li>\n<\/ol>\n<h4>Related links<\/h4>\n<ul>\n<li><a href=\"https:\/\/www.youtube.com\/watch?v=w002dYaP9mw\">Azure Friday &#8211; Azure Functions with AAD support<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-bindings-cosmosdb-v2-trigger?tabs=in-process%2Cextensionv4&amp;pivots=programming-language-csharp\">Azure Functions trigger for Cosmos DB samples<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-bindings-cosmosdb-v2-output?tabs=in-process%2Cextensionv4&amp;pivots=programming-language-csharp\">Azure Functions output binding for Cosmos DB samples<\/a><\/li>\n<li><a href=\"https:\/\/learn.microsoft.com\/azure\/azure-functions\/functions-bindings-cosmosdb-v2-output?tabs=in-process%2Cextensionv4&amp;pivots=programming-language-csharp#grant-permission-to-the-identity\">Minimum required permissions for RBAC and Azure Functions<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The Azure Cosmos DB Azure Functions extension version 4\u00a0is now GA, and it packs a load of improvements and new features. The goal of this post is to go deep into the technical details about the extension changes and guide you to take advantage of them. Source code As a starting point, everything I&#8217;ll be [&hellip;]<\/p>\n","protected":false},"author":9477,"featured_media":5299,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,644],"tags":[],"class_list":["post-5170","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-core-sql-api","category-change-feed"],"acf":[],"blog_post_summary":"<p>The Azure Cosmos DB Azure Functions extension version 4\u00a0is now GA, and it packs a load of improvements and new features. The goal of this post is to go deep into the technical details about the extension changes and guide you to take advantage of them. Source code As a starting point, everything I&#8217;ll be [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/5170","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/users\/9477"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/comments?post=5170"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/5170\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media\/5299"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media?parent=5170"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/categories?post=5170"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/tags?post=5170"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}