{"id":16703,"date":"2026-07-09T00:00:00","date_gmt":"2026-07-09T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ise\/?p=16703"},"modified":"2026-07-09T05:41:09","modified_gmt":"2026-07-09T12:41:09","slug":"from_noisy_queries_to_precise_frames","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/from_noisy_queries_to_precise_frames\/","title":{"rendered":"From Noisy Queries to Precise Frames: Query Decomposition for Media Asset Search"},"content":{"rendered":"

Introduction: The Problem<\/h2>\n

Media and entertainment teams work with massive asset catalogs for films and TV series. Each asset often includes structured metadata such as episode ID, scene ID, and shot ID, along with the visual content itself. Teams want a unified search experience that understands both metadata constraints and natural language intent.<\/p>\n

The problem appears when users provide both types of information in one query, for example:<\/p>\n

Two soldiers facing off across a battlefield in episode 20, scene 5, shot 2<\/code><\/pre>\n
If we embed this raw query directly for semantic retrieval, two issues appear:<\/p>\n
    \n
  1. Metadata tokens dilute visual intent in the embedding representation.<\/li>\n
  2. Retrieval can return visually similar assets from the wrong episode, scene, or shot.<\/li>\n<\/ol>\n
    This gap is common in real workflows. Editors, archivists, and production teams often know part of the context, but they still describe the visual moment in natural language. Query decomposition solves this by splitting one noisy query into two useful parts: structured filters and semantic intent.<\/p>\n
    The Journey: Our Approach and Solution<\/h2>\n
    In production, this works as a query pre-processing layer between user input and retrieval: the parser first identifies grounded metadata signals such as episode, scene, and shot, then removes those tokens to generate a clean sub-query that captures only the visual intent. The metadata fields are converted into strict filter clauses to narrow the candidate pool, while the sub-query is embedded for semantic ranking within that filtered set. This two-path retrieval flow prevents context drift, reduces false positives from adjacent scenes, and still preserves semantic flexibility when users describe actions, objects, or mood in free-form language.<\/p>\n
    Approach<\/h3>\n
    Our approach decomposes each user query into:<\/p>\n
      \n
    1. Structured filters: episode, scene, shot (and optionally characters)<\/li>\n
    2. A clean sub-query: natural language description with metadata removed<\/li>\n<\/ol>\n
      We then run hybrid retrieval:<\/p>\n
        \n
      1. Apply filters as pre-conditions in metadata search.<\/li>\n
      2. Use the sub-query for semantic or vector retrieval.<\/li>\n
      3. Rank merged candidates with a hybrid strategy.<\/li>\n<\/ol>\n
        \"Overall<\/p>\n
        At a high level, the decomposition pipeline follows three steps:<\/p>\n
          \n
        1. Parse query text with any one of the approaches, i.e. Regex or LLM or fine-tuned Entity extraction.<\/li>\n
        2. Extract explicit metadata fields.<\/li>\n
        3. Rewrite the residual visual intent as a sub-query.<\/li>\n<\/ol>\n
          This design improves precision through filtering and improves recall through semantic matching.<\/p>\n
          Data available<\/h3>\n
          For this use case, we evaluated approximately 500 user queries with mixed structure:<\/p>\n
            \n
          1. Queries with full contextual metadata (episode, scene, and shot)<\/li>\n
          2. Queries with partial contextual metadata<\/li>\n
          3. Queries with no explicit contextual metadata<\/li>\n<\/ol>\n
            Ground-truth template<\/h3>\n
            Domain experts labeled the 500 query set using this template:<\/p>\n\n\n\n\n\n
            Query<\/th>\nShot-id<\/th>\nScene-id<\/th>\nEpisode-id<\/th>\nSub-query<\/th>\n<\/tr>\n<\/thead>\n
            User input query<\/td>\nExtracted shot value<\/td>\nExtracted scene value<\/td>\nExtracted episode value<\/td>\nMetadata-free semantic query<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
            Sample Queries<\/h3>\n\n\n\n\n\n\n\n\n\n
            Query<\/th>\nEpisode ID<\/th>\nScene ID<\/th>\nShot ID<\/th>\nSub-query<\/th>\n<\/tr>\n<\/thead>\n
            Two astronauts repairing a satellite in episode 4 scene 3 shot 11<\/td>\n4<\/td>\n3<\/td>\n11<\/td>\nTwo astronauts repairing a satellite<\/td>\n<\/tr>\n
            A chef plating dessert in scene 9 shot 2<\/td>\nN\/A<\/td>\n9<\/td>\n2<\/td>\nA chef plating dessert<\/td>\n<\/tr>\n
            Crowd celebrating at a city marathon finish line<\/td>\nN\/A<\/td>\nN\/A<\/td>\nN\/A<\/td>\nCrowd celebrating at a city marathon finish line<\/td>\n<\/tr>\n
            Detective opening an old letter in episode 2<\/td>\n2<\/td>\nN\/A<\/td>\nN\/A<\/td>\nDetective opening an old letter<\/td>\n<\/tr>\n
            Family watching fireworks by a river at night<\/td>\nN\/A<\/td>\nN\/A<\/td>\nN\/A<\/td>\nFamily watching fireworks by a river at night<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
            Experiments & Evaluation<\/h2>\n
            Regex<\/code> vs LLM<\/code> vs Fine-Tuned Entity Extraction<\/code><\/h3>\n
            Each extraction approach comes with different trade-offs depending on your query patterns, budget, and maintainability needs:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
            Criterion<\/th>\nRegex<\/th>\nLLM (GPT-4.1-mini \/ GPT-4.1)<\/th>\nFine-tuned Entity Extraction<\/th>\n<\/tr>\n<\/thead>\n
            Latency<\/td>\nZero – no network call<\/td>\n4\u20135 seconds per query<\/td>\nNear-zero after deployment<\/td>\n<\/tr>\n
            Cost<\/td>\nFree<\/td>\nPer-token API cost<\/td>\nOne-time training + hosting cost<\/td>\n<\/tr>\n
            Sub-query rewriting<\/td>\nManual \/ rule-based<\/td>\nNatural and fluent<\/td>\nManual \/ rule-based – entity extraction only<\/td>\n<\/tr>\n
            Typo & variation tolerance<\/td>\nLow – brittle on irregular input<\/td>\nHigh – generalises across phrasing<\/td>\nMedium – bounded by training distribution<\/td>\n<\/tr>\n
            Extensibility to new fields<\/td>\nRequires new rules per field<\/td>\nPrompt update only<\/td>\nRequires re-training or fine-tuning<\/td>\n<\/tr>\n
            Operational complexity<\/td>\nVery low<\/td>\nLow (API)<\/td>\nHigh – model lifecycle, infra, versioning<\/td>\n<\/tr>\n
            Best suited for<\/td>\nConsistent, well-structured input<\/td>\nMixed or ambiguous input<\/td>\nHigh-volume production with domain shift<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
            Experiment Setup and Variants<\/h3>\n
            We tested four decomposition modes:<\/p>\n
            \"Experiment<\/p>\n
            Evaluation<\/h3>\n
            We evaluated decomposition quality and retrieval impact using three metric groups:<\/p>\n
              \n
            1. Filter extraction accuracy: exact match for episode, scene, and shot<\/li>\n
            2. Sub-query quality: ROUGE-L and BLEU against labeled rewrites<\/li>\n
            3. Retrieval effectiveness: Recall@K (K=5)<\/li>\n<\/ol>\n
              The Destination: Outcomes and Learnings<\/h2>\n
              Decomposition Quality<\/h3>\n\n\n\n\n\n\n\n\n\n
              Method<\/th>\nEpisode match<\/th>\nScene match<\/th>\nShot match<\/th>\nROUGE-L<\/th>\nBLEU<\/th>\nLatency<\/th>\n<\/tr>\n<\/thead>\n
              GPT-4.1 (zero-shot)<\/td>\n100%<\/td>\n99.7%<\/td>\n100%<\/td>\n1.00<\/td>\n0.86<\/td>\n5s<\/td>\n<\/tr>\n
              GPT-4.1-mini (zero-shot)<\/td>\n100%<\/td>\n99.7%<\/td>\n100%<\/td>\n1.00<\/td>\n0.85<\/td>\n4s<\/td>\n<\/tr>\n
              GPT-4.1 (few-shot)<\/td>\n100%<\/td>\n100%<\/td>\n100%<\/td>\n1.00<\/td>\n0.86<\/td>\n5s<\/td>\n<\/tr>\n
              GPT-4.1-mini (few-shot)<\/td>\n100%<\/td>\n100%<\/td>\n100%<\/td>\n1.00<\/td>\n0.85<\/td>\n4s<\/td>\n<\/tr>\n
              Regex<\/td>\n100%<\/td>\n100%<\/td>\n100%<\/td>\n0.97<\/td>\n0.79<\/td>\n0s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
              Token Consumption<\/h3>\n\n\n\n\n\n\n\n
              Variant<\/th>\nAvg input tokens<\/th>\nAvg output tokens<\/th>\n<\/tr>\n<\/thead>\n
              Regex<\/td>\n0<\/td>\n0<\/td>\n<\/tr>\n
              zero-shot<\/td>\n700<\/td>\n35<\/td>\n<\/tr>\n
              few-shot<\/td>\n1053<\/td>\n35<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
              Retrieval Effectiveness (Recall@K, K=5)<\/h3>\n
                \n
              1. Query-only baseline: about 27%<\/li>\n
              2. Sub-query only: about 31.8% to 32.2%<\/li>\n
              3. Filters + sub-query: best overall, up to 49.4%<\/li>\n<\/ol>\n
                Detailed highlights:<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n
                Method<\/th>\nQuery only<\/th>\nSub-query only<\/th>\nFilters + sub-query<\/th>\n<\/tr>\n<\/thead>\n
                Regex<\/td>\n27%<\/td>\n32.2%<\/td>\n49.4%<\/td>\n<\/tr>\n
                GPT-4.1-mini (zero-shot)<\/td>\n27%<\/td>\n32.2%<\/td>\n46.2%<\/td>\n<\/tr>\n
                GPT-4.1-mini (few-shot)<\/td>\n27%<\/td>\n31.8%<\/td>\n49%<\/td>\n<\/tr>\n
                GPT-4.1 (zero-shot)<\/td>\n27%<\/td>\n31.8%<\/td>\n43.6%<\/td>\n<\/tr>\n
                GPT-4.1 (few-shot)<\/td>\n27%<\/td>\n31.8%<\/td>\n49%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
                Key observations<\/h3>\n
                  \n
                1. Few-shot closed the Scene ID gap seen in zero-shot.<\/li>\n
                2. Regex matched or exceeded LLM quality for structural extraction.<\/li>\n
                3. Filters plus semantic sub-query produced the strongest retrieval outcome.<\/li>\n<\/ol>\n
                  Why and When to Use an LLM<\/h3>\n
                  Regex is the best choice when queries are structured and format patterns are stable.<\/p>\n
                  LLMs become valuable when you need:<\/p>\n
                    \n
                  1. Robust sub-query rewriting for semantic search quality<\/li>\n
                  2. Tolerance for typos and natural phrasing variations<\/li>\n
                  3. Extensibility to future fields such as character names or location entities<\/li>\n<\/ol>\n
                    A practical strategy is to use Regex first, then invoke an LLM fallback (say, GPT-4.1-mini) for edge cases that fail rule coverage.<\/p>\n
                    Edge-Case Comparison: Where LLM Wins and Where Regex Leads<\/h3>\n
                    The examples below show why a hybrid strategy works best in production. LLM-based decomposition excelled at preserving natural phrasing and correctly separating metadata from intent in ambiguous patterns, while Regex remained strongest on speed, determinism, and zero-cost extraction for stable formats.<\/p>\n\n\n\n\n\n\n\n
                    Query<\/th>\nIdeal output<\/th>\nRegex output<\/th>\nLLM output<\/th>\nWhat this shows<\/th>\n<\/tr>\n<\/thead>\n
                    Scene 3 A frontal pose of warriors with a hill background<\/td>\n– scene: 3\n– sub-query: A frontal pose of warriors with a hill background<\/td>\n– scene: 3A\n– sub-query: frontal pose of warriors with a hill background<\/td>\n– scene: 3\n– sub-query: A frontal pose of warriors with a hill background<\/td>\nLLM handled boundary ambiguity better by keeping 3<\/code> as scene and preserving the complete semantic phrase.<\/td>\n<\/tr>\n
                    Frontal Profile of Spiderman and Superman in scene 10A<\/td>\n– scene: 10A\n– sub-query: Frontal Profile of Spiderman and Superman<\/td>\n– scene: 10A\n– sub-query: Frontal Profile of Spiderman and Superman in<\/td>\n– scene: 10A\n– sub-query: Frontal Profile of Spiderman and Superman<\/td>\nLLM produced cleaner rewrite quality by removing trailing connector tokens after metadata extraction.<\/td>\n<\/tr>\n
                    A wide shot of cyclists crossing a bridge in scnee 12<\/td>\n– scene: 12\n– sub-query: A wide shot of cyclists crossing a bridge<\/td>\n– scene: N\/A\n– sub-query: A wide shot of cyclists crossing a bridge in scnee 12<\/td>\n– scene: 12\n– sub-query: A wide shot of cyclists crossing a bridge<\/td>\nLLM tolerated the typo in scene<\/code> and still extracted the correct filter, while Regex missed the malformed token pattern.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n\n\n\n
                    Regex strengths<\/th>\nLLM strengths<\/th>\n<\/tr>\n<\/thead>\n
                    Deterministic behavior for known templates<\/td>\nBetter handling of ambiguous token boundaries<\/td>\n<\/tr>\n
                    Zero latency and zero token cost<\/td>\nMore fluent sub-query rewriting<\/td>\n<\/tr>\n
                    Easier debugging for rule failures<\/td>\nBetter resilience to natural phrasing variation<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
                    Conclusion<\/h2>\n
                    Query decomposition is not an optional optimization for media asset retrieval. It is a foundational step when user queries mix contextual metadata with free-form visual descriptions.<\/p>\n
                    Our evaluation shows that separating metadata filters from semantic intent nearly doubles retrieval performance versus raw semantic search alone, moving the Recall@K (K=5) from 27% to around 49%.<\/p>\n
                    For this use case, Regex is the preferred default because it delivers perfect structural extraction with zero latency and zero token cost. When coverage needs to expand to more ambiguous input styles, GPT-4.1-mini with prompt v2 provides the best cost-to-quality fallback.<\/p>\n
                    Thanks<\/h2>\n
                    A special thanks to Mark D’Souza<\/a>, Prabal Deb<\/a>, Preethi Natarajan<\/a>, Priya Bhimjyani<\/a> for their valuable input.<\/p>\n
                    The featured image was generated using Copilot. Terms can be found here<\/a>.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"
                    How query decomposition separates metadata filters from visual intent to significantly improve media asset retrieval quality.<\/p>\n","protected":false},"author":193956,"featured_media":16704,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,1,17,19],"tags":[3400],"class_list":["post-16703","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cognitive-services","category-cse","category-frameworks","category-machine-learning","tag-ise"],"acf":[],"blog_post_summary":"
                    How query decomposition separates metadata filters from visual intent to significantly improve media asset retrieval quality.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16703","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/193956"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=16703"}],"version-history":[{"count":1,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16703\/revisions"}],"predecessor-version":[{"id":16705,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16703\/revisions\/16705"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/16704"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=16703"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=16703"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=16703"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}