{"id":5913,"date":"2023-03-30T06:00:49","date_gmt":"2023-03-30T13:00:49","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=5913"},"modified":"2023-04-04T09:07:48","modified_gmt":"2023-04-04T16:07:48","slug":"now-in-preview-change-data-capture-cdc-with-azure-cosmos-db-analytical-store","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/now-in-preview-change-data-capture-cdc-with-azure-cosmos-db-analytical-store\/","title":{"rendered":"Now in preview \u2013 Change Data Capture (CDC) with Azure Cosmos DB analytical store"},"content":{"rendered":"<p><a class=\"Hyperlink SCXW55975736 BCX8\" href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/analytical-store-introduction\" target=\"_blank\" rel=\"noreferrer noopener\"><span class=\"FieldRange SCXW55975736 BCX8\"><span class=\"TextRun Highlight SCXW55975736 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun CommentStart SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\" data-ccp-charstyle-defn=\"{&quot;ObjectId&quot;:&quot;98df8e5e-1b39-44f4-84e1-02f7d188c012|6&quot;,&quot;ClassId&quot;:1073872969,&quot;Properties&quot;:[469775450,&quot;normaltextrun&quot;,201340122,&quot;1&quot;,134233614,&quot;true&quot;,469778129,&quot;normaltextrun&quot;,335572020,&quot;1&quot;,469778324,&quot;Default Paragraph Font&quot;]}\">Azure Cosmos DB analytical store<\/span><\/span><\/span><\/a><span class=\"TextRun SCXW55975736 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"auto\"> <span class=\"NormalTextRun SCXW55975736 BCX8\">now <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">supports <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">Change Data Capture (<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">CDC)<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">, <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">for <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">Azure Cosmos DB <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">API <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">for <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">NoSQL and <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">Azure Cosmos DB API for <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">MongoDB<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">.<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\"> This capability, available in public previe<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">w<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">, <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">allows you to <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">efficiently <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">consume <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">a continuous and <\/span><span class=\"NormalTextRun CommentStart SCXW55975736 BCX8\">incremental<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\"> feed of changed<\/span> <span class=\"NormalTextRun SCXW55975736 BCX8\">(<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">inserted, <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">updated,<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\"> and <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">deleted<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">)<\/span> <span class=\"NormalTextRun SCXW55975736 BCX8\">data <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">from <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">analytical store. <\/span><\/span><span class=\"TextRun Highlight SCXW55975736 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">CDC <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">is <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">seamlessly<\/span> <span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">integrat<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">ed <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">with Azure Synapse and Azure Data <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">F<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">actory<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">, <\/span><span class=\"NormalTextRun CommentStart SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">provid<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\" data-ccp-charstyle=\"normaltextrun\">ing <\/span><\/span><span class=\"TextRun SCXW55975736 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-contrast=\"none\"><span class=\"NormalTextRun SCXW55975736 BCX8\">a scalable no-code experience for high data <\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">volume.<\/span> <span class=\"NormalTextRun SCXW55975736 BCX8\">As CDC<\/span> <span class=\"NormalTextRun SCXW55975736 BCX8\">is based on analytical <\/span><span class=\"NormalTextRun ContextualSpellingAndGrammarErrorV2Themed SCXW55975736 BCX8\">store<\/span><span class=\"NormalTextRun SCXW55975736 BCX8\">, it <a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/analytical-store-introduction#decoupled-performance-for-analytical-workloads\">does not consume provisioned RUs, does not affect the performance of your transactional workloads, provides lower latency and has lower TCO<\/a>.\u00a0<\/span><\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1.jpg\"><img decoding=\"async\" class=\"aligncenter wp-image-5955\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1.jpg\" alt=\"CDC with Azure Cosmos DB analytical store\" width=\"880\" height=\"415\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1.jpg 2419w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1-300x142.jpg 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1-1024x483.jpg 1024w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1-768x362.jpg 768w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1-1536x725.jpg 1536w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDC-image-1-2048x966.jpg 2048w\" sizes=\"(max-width: 880px) 100vw, 880px\" \/><\/a><\/p>\n<h4 style=\"text-align: center;\" aria-level=\"3\"><span style=\"font-size: 10pt;\"><a class=\"Hyperlink SCXP30479482 BCX8\" href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/data-factory\/data-flow-sink#supported-sinks\" target=\"_blank\" rel=\"noopener noreferrer\"><span class=\"TextRun Underlined SCXP30479482 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-scheme-color=\"@0000FF,,\" data-usefontface=\"true\" data-contrast=\"none\"><span class=\"NormalTextRun SCXP30479482 BCX8\">Click here<\/span><\/span><\/a><span class=\"TextRun SCXP30479482 BCX8\" lang=\"EN-US\" xml:lang=\"EN-US\" data-usefontface=\"true\" data-contrast=\"none\"><span class=\"NormalTextRun SCXP30479482 BCX8\">\u00a0for supported sink types on Mapping Data Flow.<\/span><\/span><\/span><\/h4>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Capabilities<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"none\">In addition to providing incremental data feed from analytical store to diverse targets, CDC supports the following capabilities.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">\n<h5>Support for applying filters, projections, and transformations on the change feed via source query<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">You can optionally use a source query to specify filter(s), projection(s), and transformation(s) which would all be pushed down to the analytical store. Below is a sample source-query that would only capture incremental records with <\/span><span data-contrast=\"none\">Category = <\/span><span data-contrast=\"none\">&#8216;<\/span><span data-contrast=\"none\">Urban<\/span><span data-contrast=\"none\">&#8216;<\/span><span data-contrast=\"none\">, project only a subset of fields and apply a simple transformation.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:1,&quot;335559685&quot;:720,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<pre class=\"prettyprint language-sql\" style=\"padding-left: 80px;\"><code class=\"language-sql\">    select ProductId, Product, Segment,\r\n    concat(Manufacturer, \u2018-\u2018, Category) as ManufactCategory\r\n    from c\u00a0\u00a0\r\n    where Category = \u2018Urban\u2019 <\/code><\/pre>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"1\">\n<h5>Support for capturing deletes and intermediate updates<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">Analytical store CDC captures deleted records and intermediate updates. The captured deletes and updates can be applied on sinks that support delete and update operations. The {_rid} value uniquely identifies the records and so by specifying {_rid} as key column on the sink side, the updates and deletes would be reflected on the sink.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"3\" data-aria-level=\"1\">\n<h5>Filter change feed for a specific type of operation (Insert | Update | Delete | TTL)<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">You can filter the CDC feed for a specific type of operation. For example, you have the option to selectively capture the Insert and update operations only, thereby ignoring the user-delete and TTL-delete operations.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"4\" data-aria-level=\"1\">\n<h5>Support for schema alterations, flattening, row modifier transformations and partitioning<span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"auto\">In addition to specifying filters, projections and transformations via source query, you can also perform advanced schema operations such as <\/span><a href=\"https:\/\/learn.microsoft.com\/azure\/data-factory\/data-flow-flatten\"><span data-contrast=\"none\">flattening,<\/span><\/a><span data-contrast=\"auto\"> applying <\/span><a href=\"https:\/\/learn.microsoft.com\/azure\/data-factory\/data-flow-transformation-overview\"><span data-contrast=\"none\">advanced row modifier operations<\/span><\/a><span data-contrast=\"auto\"> and dynamically partitioning the data based on the given key. <\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335551550&quot;:6,&quot;335551620&quot;:6,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"5\" data-aria-level=\"1\">\n<h5>Efficient incremental data capture with internally managed checkpoints<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">Each change in Cosmos DB container appears exactly once in the CDC feed, and the checkpoints are managed internally for you. <\/span><span data-contrast=\"none\">This helps to address the below disadvantages of the common pattern of using custom checkpoints based on the \u201c_ts\u201d value: <\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"2\"><span data-contrast=\"none\">The \u201c_ts\u201d filter is applied against the data files which does not always guarantee minimal data scan. The internally managed GLSN based checkpoints in the new CDC capability ensure that the incremental data identification is done, just based on the metadata and so guarantees minimal data scanning in each stream.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"6\" data-aria-level=\"1\"><span data-contrast=\"none\">The analytical store sync process does not guarantee \u201c_ts\u201d based ordering which means that there could be cases where an incremental record\u2019s \u201c_ts\u201d is lesser than the last checkpointed \u201c_ts\u201d and could be missed out in the incremental stream. The new CDC does not consider \u201c_ts\u201d to identify the incremental records and thus guarantees that none of the incremental records are missed.\u00a0<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">With CDC, there\u2019s no limitation around the fixed data retention period for which changes are available. Multiple change feeds on the same container can be consumed simultaneously. Changes can be synchronized from \u201cthe Beginning\u201d or \u201cfrom a given timestamp\u201d or \u201cfrom now\u201d.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"34\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"7\" data-aria-level=\"1\">\n<h5>Throughput isolation, lower latency and lower TCO<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"none\">Operations on Cosmos DB analytical store do not consume the provisioned RUs and so do not impact your transactional workloads. CDC with analytical store also has lower latency and lower TCO, compared to using ChangeFeed on transactional store. The lower latency is attributed to analytical store enabling better parallelism for data processing and reduces the overall TCO enabling you to drive cost efficiencies.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"34\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"8\" data-aria-level=\"1\">\n<h5>No-code, low-touch, non-intrusive end-to-end integration<\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"auto\">The seamless and native integration of analytical store CDC with Azure Synapse and Azure Data Factory provides the no-code, low-touch experience.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">Scenarios<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h4>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"34\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"9\" data-aria-level=\"1\">\n<h5><span data-contrast=\"none\">Consuming incremental data from Cosmos DB<\/span><\/h5>\n<p>You can consider using analytical store CDC, if you are currently using or planning to use below:\u00a0\u00a0<span style=\"font-size: 1rem; text-align: var(--bs-body-text-align);\" data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<ul>\n<li data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"2\" data-aria-level=\"2\"><span data-contrast=\"auto\">Incremental data capture using Azure Data Factory Data Flow or Copy activity<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"3\" data-aria-level=\"2\"><span data-contrast=\"none\">One time batch processing using Azure Data Factory\u202f<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/li>\n<li data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"4\" data-aria-level=\"2\"><span data-contrast=\"none\">Streaming Cosmos DB data<\/span><\/li>\n<li data-leveltext=\"o\" data-font=\"Courier New\" data-listid=\"33\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:1440,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Courier New&quot;,&quot;469769242&quot;:[9675],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;o&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"4\" data-aria-level=\"2\">Capturing deletes, intermediate changes, applying filters or projections or transformations on Cosmos DB Data<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"list-style-type: none;\"><div class=\"alert alert-info\"><p class=\"alert-divider\"><i class=\"fabric-icon fabric-icon--Info\"><\/i><strong>Analytical store sync latency<\/strong><\/p>Note that analytical store has up to 2 min latency to sync transactional store data<\/div><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"34\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559684&quot;:-2,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" aria-setsize=\"-1\" data-aria-posinset=\"1\" data-aria-level=\"1\">\n<h5><span data-contrast=\"none\">Incremental feed to analytical platform of your choice<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h5>\n<\/li>\n<\/ul>\n<p style=\"padding-left: 40px;\"><span data-contrast=\"auto\">Change data capture capability enables end-to-end analytical story providing you the flexibility to write Cosmos DB data to any of the <\/span><a href=\"https:\/\/learn.microsoft.com\/en-us\/azure\/data-factory\/data-flow-sink#supported-sinks\"><span data-contrast=\"none\">supported sink types<\/span><\/a><span data-contrast=\"auto\">. It also enables you to bring Cosmos DB data into a centralized data lake and join with data from diverse data sources. You can flatten the data, partition it and apply more transformations either in Azure Synapse or Azure Data Factory.<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559685&quot;:720,&quot;335559739&quot;:160,&quot;335559740&quot;:259}\">\u00a0<\/span><\/p>\n<p><center><iframe src=\"\/\/www.youtube.com\/embed\/0e0b9gjtzv4\" width=\"560\" height=\"314\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/center><\/p>\n<h4 aria-level=\"3\"><span data-contrast=\"none\">\nSetting up CDC in analytical store<\/span><span data-ccp-props=\"{&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;201341983&quot;:0,&quot;335559738&quot;:40,&quot;335559739&quot;:0,&quot;335559740&quot;:259}\">\u00a0<\/span><\/h4>\n<p><span data-contrast=\"none\">You can consume incremental analytical store data from a Cosmos DB container using either Azure Synapse or Azure Data Factory, once the <\/span><a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/configure-synapse-link#enable-synapse-link\"><span data-contrast=\"none\">Cosmos DB account has been enabled for Synapse Link<\/span><\/a><span data-contrast=\"none\"> and analytical store has been enabled on a <\/span><a href=\"https:\/\/learn.microsoft.com\/azure\/cosmos-db\/configure-synapse-link#update-analytical-ttl\"><span data-contrast=\"none\">new container<\/span><\/a><span data-contrast=\"none\"> or an <\/span><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/azure-synapse-link-existing-containers-and-power-bi-integration\/#azure-synapse-link-for-existing-azure-cosmos-containers\"><span data-contrast=\"none\">existing container<\/span><\/a><span data-contrast=\"none\">.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"none\">On the Azure Synapse Data flow or on the Azure Data factory Data flow, choose the Inline dataset type as \u201cAzure Cosmos DB for NoSQL\u201d and Store type as \u201cAnalytical\u201d, as seen below.\u00a0<\/span><span data-ccp-props=\"{&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup.png\"><img decoding=\"async\" class=\"aligncenter wp-image-5919\" src=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup.png\" alt=\"CDC Setup\" width=\"605\" height=\"607\" srcset=\"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup.png 1058w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-300x300.png 300w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-1021x1024.png 1021w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-150x150.png 150w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-768x770.png 768w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-24x24.png 24w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-48x48.png 48w, https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-content\/uploads\/sites\/52\/2023\/03\/CDCSetup-96x96.png 96w\" sizes=\"(max-width: 605px) 100vw, 605px\" \/><\/a><\/p>\n<p><span data-contrast=\"auto\">Please note that the linked service interface for Azure Cosmos DB for MongoDB API is not available on Dataflow yet. However, you would be able to use your account\u2019s document endpoint with the \u201cAzure Cosmos DB for NoSQL\u201d linked service interface as a work around until the Mongo linked service is supported.\u202f<\/span><\/p>\n<p><span data-contrast=\"auto\">Eg: On a NoSQL linked service, choose \u201cEnter Manually\u201d to provide the Cosmos DB account info and use account\u2019s document endpoint (eg: https:\/\/&lt;accturi&gt;.documents.azure.com:443\/) instead of the Mongo endpoint (eg: mongodb:\/\/&lt;accturi&gt;.mongo.cosmos.azure.com:10255\/)\u202f<\/span><span data-ccp-props=\"{&quot;134233117&quot;:true,&quot;134233118&quot;:false,&quot;201341983&quot;:0,&quot;335559739&quot;:0,&quot;335559740&quot;:240}\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Azure Cosmos DB analytical store now supports Change Data Capture (CDC), for Azure Cosmos DB API for NoSQL and Azure Cosmos DB API for MongoDB. This capability, available in public preview, allows you to efficiently consume a continuous and incremental feed of changed (inserted, updated, and deleted) data from analytical store. CDC is seamlessly integrated [&hellip;]<\/p>\n","protected":false},"author":9841,"featured_media":5955,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1216,15,14,644],"tags":[],"class_list":["post-5913","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analytics","category-mongodb-api","category-core-sql-api","category-change-feed"],"acf":[],"blog_post_summary":"<p>Azure Cosmos DB analytical store now supports Change Data Capture (CDC), for Azure Cosmos DB API for NoSQL and Azure Cosmos DB API for MongoDB. This capability, available in public preview, allows you to efficiently consume a continuous and incremental feed of changed (inserted, updated, and deleted) data from analytical store. CDC is seamlessly integrated [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/5913","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/users\/9841"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/comments?post=5913"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/posts\/5913\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media\/5955"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/media?parent=5913"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/categories?post=5913"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cosmosdb\/wp-json\/wp\/v2\/tags?post=5913"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}