Real time data processing with new Azure Cosmos DB change feed modes
The Azure Cosmos DB change feed allows applications to seamlessly react to real time changes to data! It’s a persistent record of changes to items in your container in the order they occurred. Instead of writing complex code to query recently modified documents, manually track checkpointing, and handle retries for errors in processing, the change feed provides ways to manage this for you. The change feed makes scenarios like triggering notifications based on updates to your data, real-time stream processing, and data movement even easier!
Change feed modes
The change feed is available in two modes, each sharing the same core concepts with key differences. Differences include the operations captured in the feed, metadata available for each change, and the retention period of changes. Multiple applications can simultaneously read the change feed for the same container in different modes to fit the requirements of your workload.
Latest version mode
If you’ve used change feed before, you’ve used latest version mode. Latest version mode is widely used to build reactive applications and was previously the only way to read change feed. This mode captures changes from insert and update operations. It always gives you the latest version of all items in the container. Documents aren’t available in the feed once they’re deleted, and you won’t get a record of their deletion in the change feed.
In latest version mode you can process the change feed going back to the beginning of the container. Changes will always be available in the feed if there are still items in the container.
New: All versions and deletes mode (Public Preview)
All versions and deletes change feed mode solves some of the key gaps in latest version mode! Every change to items in your container appears in the change feed when using this mode- including when multiple changes to a single item occur between change feed reads and when an item is deleted. Deletes from both explicit delete operations and TTL expirations are captured. Previously, there was no way to be notified when an item expired due to TTL, even if you used the “soft delete” pattern. This mode gives you a full view of the state of your container without any custom logic or soft deletes.
In order to read the change feed in all versions and deletes mode, continuous backups must be enabled on your Azure Cosmos DB account. The retention period of all versions and deletes change feed depends on the retention period configured for continuous backups. You can process and reprocess changes in this mode as long as the changes occurred within the retention period.
The change feed now serves more use cases
Regardless of the mode you read it in, the Azure Cosmos DB change feed allows for efficient processing of items that have changed in a container. Latest version mode may be the best fit for your workload in many scenarios, especially if you need to process changes from the beginning of your container. In addition to the core use cases for change feed, all versions and deletes mode enhances many patterns like event sourcing and data movement, as well as enables new ones like auditing.
Some use cases that are a great fit for all versions and deletes mode are:
- Real-time transfer of data between two locations without having to implement a soft delete
- Triggering logic based on all changes if multiple change operations for a given item occur between change feed reads
- Triggering alerts on delete operations
- The ability to process item creates, updates, and deletes differently based on operation type
Sign up for the all versions and deletes mode Public Preview
Enroll in the preview for all versions and deletes change feed mode via the Preview Features page in the Azure Portal. Search for the AllVersionsAndDeletesChangeFeed feature to register. 
After signing up for the preview, expect a response to the email you used to sign up within about a week. Once approved, you’ll need to ensure that continuous backups are enabled on your Azure Cosmos DB account before you can read the change feed in all versions and deletes mode.
Get started using all versions and deletes mode
After being enrolled in the preview and configuring continuous backups, you’re ready to get started using all versions and deletes mode! During the preview you can read the change feed using the change feed processor in Java, or using the pull model in either .NET or Java.
Here’s a sample change feed processor implementation in Java. The full sample can be found on GitHub.
ChangeFeedProcessor changeFeedProcessorInstance = new ChangeFeedProcessorBuilder()
.hostName(hostName)
.options(options)
.feedContainer(feedContainer)
.leaseContainer(leaseContainer)
.handleAllVersionsAndDeletesChanges((List<ChangeFeedProcessorItem> changeFeedProcessorItems) -> {
logger.info("--->handleAllVersionsAndDeletesChanges() START");
for (ChangeFeedProcessorItem item : changeFeedProcessorItems) {
try {
// All versions and deletes change feed items are in the form of ChangeFeedProcessorItem
// You can call the different getters for different states, as shown below. Different operations populate each state differently.
logger.info("---->CURRENT ITEM VERSION RECEIVED: {}", item.getCurrent()); // Populated for create and replace operations.
logger.info("---->PREVIOUS ITEM VERSION RECEIVED: {}", item.getPrevious()); // Populated for delete operations.
logger.info("---->METADATA RECEIVED: {}", item.getChangeFeedMetaData()); // Populated all operations.
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
logger.info("--->handleAllVersionsAndDeletesChanges() END");
})
.buildChangeFeedProcessor();
Learn more
- Working with the change feed – Azure Cosmos DB | Microsoft Learn
- Reading Azure Cosmos DB change feed | Microsoft Learn
- Change feed design patterns in Azure Cosmos DB | Microsoft Learn
Get Started with Azure Cosmos DB for free
Azure Cosmos DB is a fully managed NoSQL and relational database for modern app development with SLA-backed speed and availability, automatic and instant scalability, and support for open source PostgreSQL, MongoDB and Apache Cassandra. Try Azure Cosmos DB for free here. To stay in the loop on Azure Cosmos DB updates, follow us on Twitter, YouTube, and LinkedIn.
Light
Dark
0 comments