{"id":11294,"date":"2025-11-18T01:30:43","date_gmt":"2025-11-18T09:30:43","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cosmosdb\/?p=11294"},"modified":"2025-12-03T20:15:25","modified_gmt":"2025-12-04T04:15:25","slug":"how-kraft-heinz-achieved-1400x-faster-data-lineage-with-azure-documentdbs-diskann-vector-search-and-hybrid-search","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cosmosdb\/how-kraft-heinz-achieved-1400x-faster-data-lineage-with-azure-documentdbs-diskann-vector-search-and-hybrid-search\/","title":{"rendered":"How Kraft Heinz achieved 1,400x faster data lineage with Azure DocumentDB\u2019s DiskANN vector search and hybrid search"},"content":{"rendered":"

This article is authored by Gurvinder Singh, Principal Cloud Engineer at The Kraft Heinz Company.<\/em><\/p>\n

\n

ICYMI:<\/strong> Hear how Kraft Heinz built an AI-powered data lineage solution leveraging Azure DocumentDB hybrid search and graph querying capabilities – watch the Ignite session here: Move fast, save more with MongoDB-compatible workloads on DocumentDB<\/a><\/p>\n

\n

The enterprise-scale challenge: Data lineage across 50+ interconnected systems<\/h1>\n

At Kraft Heinz, we operate one of the world\u2019s largest CPG data ecosystems with more than 200 brands and 50+ interconnected enterprise systems\u2014including SAP, Oracle, Snowflake, and Power BI\u2014connected through 1,000+ data transformation pipelines. Our data footprint stretches from ingredient sourcing to retail analytics. Every decision, whether it\u2019s optimizing a production line or forecasting demand, depends on data moving reliably through dozens of interconnected systems.<\/p>\n

With so much data in motion, even small changes can have big effects. A missed pipeline job or schema update can ripple through analytics, dashboards, and reports. That\u2019s where data lineage becomes essential. Data lineage is <\/strong>the GPS for enterprise data<\/strong>, giving teams visibility into how information flows, transforms, and impacts downstream systems.<\/p>\n

Our lineage platform applies graph principles, modeling data as nodes and relationships so dependencies can be traced at a glance. This helps data engineers pinpoint issues quickly, assess downstream impact, and make confident changes without slowing innovation. For a company operating at global scale, lineage isn\u2019t optional\u2014it\u2019s foundational to maintaining data quality and trust.<\/p>\n

The first iteration: data lineage with a third-party graph database<\/h1>\n

Our first iteration used a third-party graph database platform to build our data lineage solution. It proved powerful and delivered measurable results. Root cause analysis dropped from hours to minutes, data quality decisions accelerated, and data engineers gained full visibility into data flows.<\/p>\n

But there were some limitations that emerged over time:<\/p>\n

    \n
  1. Cost structure: Significant annual recurring licensing fees<\/li>\n
  2. No consumption-based pricing model available<\/li>\n
  3. Performance: Complex lineage graph queries with multi-level graph traversal were slow or timing out<\/li>\n
  4. Operational overhead: Separate database island outside of Azure ecosystem<\/li>\n<\/ol>\n

    In this blog, I’ll share why we chose Azure DocumentDB<\/a> for data lineage, how we rebuilt the platform on Azure, where we\u2019re heading next as we bring AI into lineage.<\/p>\n

    Solving two challenges: reducing costs and operational overhead<\/h1>\n

    We set two clear objectives and key results (OKRs): reduce annual licensing costs and eliminate operational overhead from managing a separate third-party graph database island outside of the Azure ecosystem. Working with our Microsoft account team, we selected Azure DocumentDB\u2014built natively on Azure alongside our data infrastructure.<\/p>\n

    But the real win? \u2013 The CosmosAIGraph Solution we built that took our lineage solution at enterprise scale. We will talk about it in the next section.<\/p>\n

    Graph reimagined using CosmosAIGraph<\/h1>\n

    Data lineage at Enterprise scale requires the following two fundamentally different capabilities:<\/p>\n

      \n
    1. Deterministic, Authoritative Lineage (high precision): Ability to answer the relationship question with absolute certainty<\/li>\n
    2. Semantic similarity: Ability to find datasets when keywords do not match<\/li>\n<\/ol>\n

      The CosmosAIGraph solution, as shown in the diagram below, combines both capabilities using a dual-pattern RAG architecture that integrates GraphRAG and OmniRAG. Let’s explore why dual-pattern RAG architecture is essential and how this works.<\/p>\n

      The solution has three main elements:<\/p>\n

        \n
      1. Azure DocumentDB<\/strong>: Persistent store for data-lineage data, edges, nodes, high dimensional vector embeddings, and session data<\/li>\n
      2. Two Microservices<\/strong>: Deployed in AKS cluster\n