{"id":3306,"date":"2021-08-13T18:02:51","date_gmt":"2021-08-14T01:02:51","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=3306"},"modified":"2021-08-18T09:31:30","modified_gmt":"2021-08-18T16:31:30","slug":"now-generally-available-azure-cosmos-db-kafka-source-sink-connectors","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/now-generally-available-azure-cosmos-db-kafka-source-sink-connectors\/","title":{"rendered":"Now Generally Available \u2013 Azure Cosmos DB Kafka Source & Sink Connectors"},"content":{"rendered":"

Source and sink Apache Kafka connectors for Azure Cosmos DB are now generally available, in collaboration with Confluent. The connectors can also be used in self-managed clusters if preferred, with no code changes needed in the application tier and controlled fully through configurations.<\/p>\n

<\/h2>\n

 <\/p>\n

Kafka Connector Use Cases<\/h2>\n

Microservice Architectures<\/h3>\n

\"Image<\/a><\/p>\n

Figure 1: Microservices communicating with one another in a database agnostic manner by leveraging Kafka<\/em><\/span><\/strong><\/p>\n

Microservice architectures involve multiple services operating independently to serve a business function. These microservices may be built on different data stores, with a variety of formats and schemas. Some may be backed by relational data stores, some on horizontally scalable NoSQL services like Azure Cosmos DB and others on analytical and warehousing stores.<\/p>\n

However, these microservices will need to communicate with one another and share data for serving additional business functions. Furthermore, data from these microservices may be needed for serving downstream functions such as analytics, aggregations and archiving among others.<\/p>\n

In the absence of a common platform like Kafka, communication pipelines between services cannot be data and format agnostic, with the need for deeply complex and intertwined dependencies. With Kafka\u2019s rich ecosystem of source and sink connectors for a plethora of database services, microservices can communicate with each another and other downstream services seamlessly, while continuing to operate independently on a database that best fits their immediate function.<\/p>\n

 <\/p>\n

Data Migration<\/h3>\n

\"Image<\/a>Figure 2: Data migration architecture leveraging Kafka as a middle-man<\/em><\/strong><\/span><\/p>\n

Data modernization efforts frequently involve migrations from to highly scalable and highly available services like Azure Cosmos DB. Tables from relational data stores are moved into Kafka topics either through CDC functionality or source Kafka connectors. Once staged, they can be converted into JSON documents through varying degrees of transformations before being ingested into Azure Cosmos DB at scale.<\/p>\n

Smaller migrations may involve a simple movement of data with minimal transformations, while the more common large scale migrations will involve additional components in the Kafka ecosystem such as Kafka Streams<\/a> and ksqlDB<\/a> in conjunction with Kafka Connect.<\/p>\n

 <\/p>\n

Source & Sink Connector Architecture<\/h2>\n

Source Connector<\/h3>\n

\"Image<\/a><\/p>\n

Figure 3: Architecture of the Kafka Source Connector for Azure Cosmos DB<\/em><\/strong><\/span><\/p>\n

The source connector for Azure Cosmos DB is built using the Java v4 client<\/a> by leveraging the inbuilt Change Feed Processor as well as the Apache Kafka Producer library.<\/p>\n

Much like a Kafka Consumer that is responsible for retrieving changes from each broker for the topic, handle offsets and periodically checkpoint progress, the Change Feed Processor for Azure Cosmos DB is responsible for picking up changes across the underlying physical partitions for the container, managing lease ownership and publishing continuation tokens (equivalent of offsets in Kafka) to track progress.<\/p>\n

With a Kafka Connect cluster configured with the Azure Cosmos DB source connector, the following operations are performed seamlessly under the covers:<\/p>\n