{"id":10261,"date":"2025-05-23T11:00:05","date_gmt":"2025-05-23T18:00:05","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=10261"},"modified":"2025-05-27T14:37:48","modified_gmt":"2025-05-27T21:37:48","slug":"confluent-cloud-releases-managed-v2-kafka-connector-for-azure-cosmos-db","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/confluent-cloud-releases-managed-v2-kafka-connector-for-azure-cosmos-db\/","title":{"rendered":"Confluent Cloud Releases Managed V2 Kafka Connector for Azure Cosmos DB"},"content":{"rendered":"<p><em><strong>This article was co-authored by Sudhindra Sheshadrivasan, Staff Product Manager at Confluent.<\/strong><\/em><\/p>\n<p>We\u2019re excited to announce the General Availability (GA) of the Confluent Cloud-managed V2 Kafka Connector for Azure Cosmos DB! This release marks a major milestone in our mission to simplify and streamline real-time data streaming from and into Azure Cosmos DB using Apache Kafka.<\/p>\n<p>The V2 connector is now production-ready and available directly from <a href=\"https:\/\/docs.confluent.io\/cloud\/current\/connectors\/index.html#supported-connectors\" target=\"_blank\" rel=\"noopener\">Confluent Cloud\u2019s connector catalog<\/a>. This managed connector allows you to seamlessly integrate Cosmos DB with your Kafka-powered event streaming architecture, without worrying about provisioning, scaling, or managing the connector infrastructure.<\/p>\n<p>Whether you\u2019re building a real-time analytics pipeline, syncing operational data between services, or enabling event-driven microservices, this connector empowers you to unlock the full potential of change data capture (CDC) and low-latency writes across your Azure Cosmos DB containers.<\/p>\n<h3><strong>Why V2 of the New Connectors?<\/strong><\/h3>\n<p>The release of version 2.0 isn\u2019t just a simple upgrade &#8211; it\u2019s a complete architectural overhaul that addresses the limitations of version 1.0 and introduces scalability, performance, and flexibility across both the source and sink connectors. Check out <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/how-to-migrate-from-kafka-connector-v1-to-v2\">migration guidance<\/a> for detailed comparisons between V1 and V2 and what to keep in mind when upgrading.<\/p>\n<h4><strong>Key motivations behind V2 include:<\/strong><\/h4>\n<ul>\n<li>Scalability Bottlenecks: In V1, every container required a separate Kafka task for CDC. V2 eliminates this constraint by allowing multiple containers to be processed by a single Kafka task, dramatically improving task efficiency and lowering operational cost.<\/li>\n<li>Improved Performance: By adopting the pull-based change feed model, the source connector reduces memory usage and thread contention\u2014leading to more stable and performant stream processing.<\/li>\n<li>Flexible Write Patterns: The sink connector now supports multiple write strategies, enabling much finer control over how data lands in Cosmos DB, especially in concurrent or distributed write scenarios.<\/li>\n<li>Better Integration with Kafka Infrastructure: Metadata management now leverages Kafka\u2019s native offset tracking, removing the dependency on Cosmos-specific lease containers and ensuring safer, more reliable checkpointing.<\/li>\n<\/ul>\n<p>In short, V2 was built from the ground up to be cloud-native, more intelligent, and aligned with Kafka best practices.<\/p>\n<h3><strong>Advantages of the New Connectors<\/strong><\/h3>\n<p>The GA version of the Confluent-managed Cosmos DB Kafka connector introduces critical enhancements that make it a powerful tool for enterprise-grade streaming and event-sourcing workloads:<\/p>\n<ul>\n<li><strong>High-Throughput Change Feed Reads<\/strong>\nThanks to container batching and pull-model support, you can ingest high volumes of changes from many containers without a proportional increase in Kafka tasks.<\/li>\n<li><strong>Advanced Sink Strategies<\/strong>\nChoose from multiple write modes including:<\/p>\n<ul>\n<li>Item Override (default)<\/li>\n<li>Create Only<\/li>\n<li>Delete<\/li>\n<li>Update If ETag Matches\nThese modes offer fine-grained control over upserts, deletes, and conditional updates &#8211; ideal for microservices architectures and stateful event handling.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Built-in Throughput Control<\/strong>\nPrevent throttling and manage RU usage more effectively with native <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/nosql\/throughput-control-spark\">throughput control<\/a>, essential for cost and performance optimization in Cosmos DB.<\/li>\n<li><strong>Integrated Metrics and Observability<\/strong>\nDeep integration with Confluent monitoring and Azure SDK metrics means you get full visibility into connector health, lag, throughput, and errors\u2014perfect for SRE teams and production troubleshooting.<\/li>\n<li><strong>Improved Security and Authentication<\/strong>\nNow supports Azure Entra ID Service Principals using client secrets.<\/li>\n<li><strong>More Reliable Metadata Handling<\/strong>\nEliminates reliance on lease containers by adopting Kafka-native offset management and introduces a low-frequency metadata topic for robust handling of partition splits and merges\u2014ensuring resilience during Cosmos DB\u2019s dynamic scaling events.<\/li>\n<\/ul>\n<p>With these capabilities, the V2 connector is now fully equipped to support high-scale, production-ready, event-driven solutions in the cloud.<\/p>\n<p><strong>\u00a0<\/strong><\/p>\n<h3>Getting started with the new connectors<\/h3>\n<p>The new connectors are very easy to setup. Before getting started on the setup itself, please note the pre-requisites:<\/p>\n<h4><strong>Azure Cosmos DB Source Prerequisites<\/strong><\/h4>\n<ol>\n<li>Authorized access to a <a href=\"https:\/\/www.confluent.io\/confluent-cloud\/\">Confluent<\/a> <a href=\"https:\/\/www.confluent.io\/confluent-cloud\/\">Cloud<\/a> cluster on Amazon Web Services (AWS), Microsoft Azure (Azure), or Google Cloud.<\/li>\n<li>The Confluent CLI installed and configured for the cluster. See <a href=\"https:\/\/docs.confluent.io\/confluent-cli\/current\/install.html\" target=\"_blank\" rel=\"noopener\">Install the Confluent CLI<\/a>.<\/li>\n<li><a href=\"https:\/\/docs.confluent.io\/cloud\/current\/get-started\/schema-registry.html#cloud-sr-config\">Schema<\/a> <a href=\"https:\/\/docs.confluent.io\/cloud\/current\/get-started\/schema-registry.html#cloud-sr-config\">Registry<\/a> must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf).<\/li>\n<li>Authorized access to read data in Azure Cosmos DB. For more information, see <a href=\"https:\/\/docs.microsoft.com\/en-us\/azure\/cosmos-db\/secure-access-to-data\" target=\"_blank\" rel=\"noopener\">Secure access to data in Azure Cosmos DB<\/a>.<\/li>\n<li>The Azure Cosmos DB is configured to use the NoSQL API.<\/li>\n<\/ol>\n<p><strong>\u00a0<\/strong><\/p>\n<h4><strong>Azure Cosmos DB Sink connector Prerequisites<\/strong><\/h4>\n<ol>\n<li>Authorized access to a <a href=\"https:\/\/www.confluent.io\/confluent-cloud\/\">Confluent<\/a> <a href=\"https:\/\/www.confluent.io\/confluent-cloud\/\">Cloud<\/a> cluster on Amazon Web Services (AWS), Microsoft Azure (Azure), or Google Cloud.<\/li>\n<li>The Confluent CLI installed and configured for the cluster. See <a href=\"https:\/\/docs.confluent.io\/confluent-cli\/current\/install.html\" target=\"_blank\" rel=\"noopener\">Install the Confluent CLI<\/a>.<\/li>\n<li><a href=\"https:\/\/docs.confluent.io\/cloud\/current\/get-started\/schema-registry.html#cloud-sr-config\">Schema<\/a> <a href=\"https:\/\/docs.confluent.io\/cloud\/current\/get-started\/schema-registry.html#cloud-sr-config\">Registry<\/a> must be enabled to use a Schema Registry-based format (for example, Avro, JSON_SR (JSON Schema), or Protobuf).<\/li>\n<li>At least one source Kafka topic must exist in your Confluent Cloud cluster before creating the sink connector.<\/li>\n<li>The Azure Cosmos DB and the Kafka cluster must be in the same region.<\/li>\n<li>The Azure Cosmos DB requires an id field in every record.<\/li>\n<\/ol>\n<h3>Connector Setup<\/h3>\n<p>Setting up of the new connectors is very similar to the regular setup experience with Confluent Cloud. Connectors can be set up via UI, CLI or API.<\/p>\n<h4><strong>Steps to setup the Azure Cosmos DB Source V2 connector.<\/strong><\/h4>\n<ol>\n<li><strong>Log in to Confluent Cloud<\/strong>:\n<ol>\n<li>Go to <a href=\"https:\/\/confluent.cloud\/\">https<\/a><a href=\"https:\/\/confluent.cloud\/\">:\/\/<\/a><a href=\"https:\/\/confluent.cloud\/\">confluent<\/a><a href=\"https:\/\/confluent.cloud\/\">.<\/a><a href=\"https:\/\/confluent.cloud\/\">cloud<\/a><a href=\"https:\/\/confluent.cloud\/\">\/<\/a> and sign in<\/li>\n<li>Select your environment and cluster<\/li>\n<\/ol>\n<\/li>\n<li><strong>Create the Connector<\/strong>:\n<ol>\n<li>Navigate to the &#8220;Connectors&#8221; tab in the left sidebar<\/li>\n<li>Click &#8220;Add connector&#8221;<\/li>\n<li>Search for &#8220;Azure Cosmos DB&#8221; and select &#8220;Azure Cosmos DB Source Connector v2&#8221;<\/li>\n<\/ol>\n<\/li>\n<li><strong>Authenticate with the kafka cluster<\/strong>\n<ol>\n<li>Authenticate via service account or an API key.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Azure Cosmos DB Connection Settings<\/strong>:\nConnect to the DB with the following:<\/p>\n<ol>\n<li>Cosmos Endpoint: The Azure Cosmos DB endpoint URL.<\/li>\n<li>Cosmos Database name: The name of your Cosmos database.<\/li>\n<li>Cosmos Connection Auth Type: MasterKey or ServicePrincipal<\/li>\n<\/ol>\n<\/li>\n<li><strong>Configuration<\/strong>:\n<ol>\n<li>Choose the appropriate output record value format (AVRO, JSON_SR, PROTOBUF)<\/li>\n<li>Setup a Topic-Container map: A comma-delimited list of Kafka topics mapped to Cosmos containers.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Sizing<\/strong>:\n<ol>\n<li>Specify the number of tasks required for the connector.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Review and Launch<\/strong>:\n<ol>\n<li>Verify all settings<\/li>\n<li>Click &#8220;Launch&#8221; to deploy the connector<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>Here is a video which details the source connector setup process:<\/p>\n<p><iframe src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2025\/05\/Cosmosdb-source-v2-demo.mp4\" width=\"1260\" height=\"720\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<p>&nbsp;<\/p>\n<h4><strong>Steps to setup the Azure Cosmos DB Sink V2 connector.<\/strong><\/h4>\n<p><strong>\u00a0<\/strong><\/p>\n<ol>\n<li><strong>Log in to Confluent Cloud<\/strong>:\n<ol>\n<li>Go to <a href=\"https:\/\/confluent.cloud\/\">https<\/a><a href=\"https:\/\/confluent.cloud\/\">:\/\/<\/a><a href=\"https:\/\/confluent.cloud\/\">confluent<\/a><a href=\"https:\/\/confluent.cloud\/\">.<\/a><a href=\"https:\/\/confluent.cloud\/\">cloud<\/a><a href=\"https:\/\/confluent.cloud\/\">\/<\/a> and sign in<\/li>\n<li>Select your environment and cluster<\/li>\n<\/ol>\n<\/li>\n<li><strong>Create the Connector<\/strong>:\n<ol>\n<li>Navigate to the &#8220;Connectors&#8221; tab in the left sidebar<\/li>\n<li>Click &#8220;Add connector&#8221;<\/li>\n<li>Search for &#8220;Azure Cosmos DB&#8221; and select &#8220;Azure Cosmos DB Sink Connector v2&#8221;<\/li>\n<\/ol>\n<\/li>\n<li><strong>Choose the right topic from the topics list<\/strong><\/li>\n<li><strong>Authenticate with the kafka cluster<\/strong>\n<ol>\n<li>Authenticate via service account or an API key.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Azure Cosmos DB Connection Settings<\/strong>:\nConnect to the DB with the following:<\/p>\n<ol>\n<li>Cosmos Endpoint: The Azure Cosmos database endpoint URL.<\/li>\n<li>Cosmos Database name: The name of your Cosmos database.<\/li>\n<li>Cosmos Connection Auth Type: MasterKey or ServicePrincipal<\/li>\n<\/ol>\n<\/li>\n<li><strong>Configuration<\/strong>:\n<ol>\n<li>Choose the appropriate input record value format (AVRO, JSON_SR, PROTOBUF)<\/li>\n<li>Specify a Topic-Container map: A comma-delimited list of Kafka topics mapped to Cosmos containers.<\/li>\n<li>Choose the right Cosmos DB Item write Strategy. Note that this defaults to ItemOverwrite.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Sizing<\/strong>:\n<ol>\n<li>Specify the number of tasks required for the connector.<\/li>\n<\/ol>\n<\/li>\n<li><strong>Review and Launch<\/strong>:\n<ol>\n<li>Verify all settings<\/li>\n<li>Click &#8220;Launch&#8221; to deploy the connector<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>Here is a video which details the sync connector setup process:<\/p>\n<p><iframe src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2025\/05\/Cosmosdb-sink-v2-demo.mp4\" width=\"1260\" height=\"720\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/p>\n<h3>Conclusion<\/h3>\n<p>The release of new fully managed Azure Cosmos DB V2 Source and Sink connectors marks a significant milestone for organizations leveraging both Confluent Kafka and Microsoft&#8217;s globally distributed Azure Cosmos DB database service. These source and sink connectors eliminate the complexity previously associated with integrating these powerful technologies, providing a streamlined path for real-time data movement between Azure Cosmos DB and Kafka ecosystems.<\/p>\n<p>By implementing these connectors, data teams can now:<\/p>\n<ul>\n<li>Capture Azure Cosmos DB change feeds directly into Confluent Kafka topics with minimal latency<\/li>\n<li>Stream data from Confluent Kafka into Azure Cosmos DB collections without custom code development<\/li>\n<li>Maintain consistency across distributed systems through configurable transformation options<\/li>\n<li>Leverage Confluent Cloud fully managed services to lower operational complexity and reduce costs<\/li>\n<\/ul>\n<p>For organizations building event-driven architectures, these connectors represent a critical building block that simplifies infrastructure while improving reliability. The ability to seamlessly connect Azure Cosmos DB&#8217;s capabilities with Kafka&#8217;s streaming prowess opens new possibilities for real-time analytics, microservices synchronization, and data replication scenarios.<\/p>\n<p>As cloud-native architectures continue to evolve, having certified, well-maintained integration points between key services becomes increasingly vital. We&#8217;re excited to see how these connectors will empower developers to build more responsive, resilient data pipelines that leverage the best of both platforms.<\/p>\n<h2 id=\"leave-a-review\"><strong>Leave a review<\/strong><button class=\"linkicon\" aria-label=\"Copy Post URL\"><\/button><\/h2>\n<p>Tell us about your Azure Cosmos DB experience! Leave a review on PeerSpot and we\u2019ll gift you $50.\u00a0<a href=\"https:\/\/peerspotdotcom.my.site.com\/proReviews\/?SalesOpportunityProduct=00kPy000004TKXJIA4&amp;productPeerspotNumber=30881&amp;CalendlyAccount=peerspot&amp;CalendlyFormLink=peerspot-product-reviews-ps-gc-vi-sf-50&amp;giftCard=50\" target=\"_blank\" rel=\"noopener\">Get started here<\/a>.<\/p>\n<h2 id=\"about-azure-cosmos-db\"><strong>About Azure Cosmos DB<\/strong><\/h2>\n<p>Azure Cosmos DB is a fully managed and serverless NoSQL and vector database for modern app development, including AI applications. With its SLA-backed speed and availability as well as instant dynamic scalability, it is ideal for real-time NoSQL and MongoDB applications that require high performance and distributed computing over massive volumes of NoSQL and vector data.<\/p>\n<p>To stay in the loop on Azure Cosmos DB updates, follow us on\u00a0<a href=\"https:\/\/twitter.com\/AzureCosmosDB\" target=\"_blank\" rel=\"noopener\">X<\/a>,\u00a0<a href=\"https:\/\/aka.ms\/AzureCosmosDBYouTube\" target=\"_blank\" rel=\"noopener\">YouTube<\/a>, and\u00a0<a href=\"https:\/\/www.linkedin.com\/company\/azure-cosmos-db\/\" target=\"_blank\" rel=\"noopener\">LinkedIn<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article was co-authored by Sudhindra Sheshadrivasan, Staff Product Manager at Confluent. We\u2019re excited to announce the General Availability (GA) of the Confluent Cloud-managed V2 Kafka Connector for Azure Cosmos DB! This release marks a major milestone in our mission to simplify and streamline real-time data streaming from and into Azure Cosmos DB using Apache [&hellip;]<\/p>\n","protected":false},"author":9387,"featured_media":10366,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,644,643,1941],"tags":[],"class_list":["post-10261","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-core-sql-api","category-change-feed","category-java-sdk","category-kafka-connectors"],"acf":[],"blog_post_summary":"<p>This article was co-authored by Sudhindra Sheshadrivasan, Staff Product Manager at Confluent. We\u2019re excited to announce the General Availability (GA) of the Confluent Cloud-managed V2 Kafka Connector for Azure Cosmos DB! This release marks a major milestone in our mission to simplify and streamline real-time data streaming from and into Azure Cosmos DB using Apache [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/10261","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/users\/9387"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/comments?post=10261"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/10261\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media\/10366"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media?parent=10261"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/categories?post=10261"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/tags?post=10261"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}