Manual tagging does not scale. We needed a way to automatically extract rich, structured metadata from images<\/strong>\u2014and critically, to trust<\/strong> the output enough to put it in a production search index.<\/p>\n The core challenge: multimodal LLMs are excellent at describing images, but they hallucinate.<\/strong> They confidently report characters not in the scene and invent props that do not exist. For a search catalog, a hallucinated tag is worse than a missing one\u2014it erodes trust in the system.<\/p>\n The question was not “can an LLM extract metadata from images?”\u2014it was “can we make the output reliable enough to trust at scale?”<\/strong><\/p><\/blockquote>\n Enrichment and embeddings are complementary. Enrichment enables faceted search and filtering. Embeddings enable semantic similarity. Use both.<\/p><\/blockquote>\n Before diving into how it works, here is what the enrichment pipeline produces. Given a scene image, the pipeline outputs structured JSON:<\/p>\n Every value in that JSON comes from a constrained set of options<\/strong>\u2014not free text. That design decision makes the entire pipeline evaluable, and it ties the rest of this post together.<\/p>\n The following diagram shows the complete flow\u2014from input image through the pipeline to validated metadata:<\/p>\n When a production team provides context\u2014“this scene has characters A, B, and C”<\/em>\u2014the model is tempted to report all three even if only one appears. Context biases the model toward expected content rather than observed content.<\/p>\n Rule: The final image is ground truth.<\/strong> Reference images and text help the model identify and name<\/em> elements\u2014but only elements it can actually see<\/em>. Everything else must be marked Think of it like a witness in court\u2014background knowledge helps identify people, but testimony must cover only what was personally observed.<\/p>\n Instead of asking the model “describe the environment”<\/em> (free text, impossible to evaluate), the pipeline asks “classify the environment into one of these categories”<\/em>:<\/p>\n Why this matters:<\/strong><\/p>\n The model’s output is enforced via JSON schema mode<\/strong> ( The Raw model output\u2014even with structured JSON enforced\u2014is never trusted directly. A 10-stage validation pipeline<\/strong> acts as an independent safety net:<\/p>\n Stages 6, 7, and 8 are the hallucination guards<\/strong>\u2014they unconditionally remove any character or prop marked Items marked Analogy:<\/strong> The prompt layer is like training an employee to be accurate. The validator layer is like having a supervisor check every report. You need both.<\/p><\/blockquote>\n The human-readable descriptions (“A torch-lit indoor palace at night with a warrior protagonist”<\/em>) are not generated by the LLM<\/strong>. They are composed deterministically from the validated structured fields\u2014so descriptions always match the structured data. No contradictions, no drift.<\/p>\n Because every output field is constrained to a finite set of values, evaluation becomes a classification problem<\/strong>\u2014not a subjective comparison. We proposed a ground truth evaluation template:<\/p>\n The template also captures edge cases<\/strong>: images with no characters (expect an empty list), ambiguous lighting (expect Key insight:<\/strong> Constrained output makes evaluation tractable<\/em>. Enum-constrained evaluation is a standard classification problem.<\/p><\/blockquote>\n The enrichment pipeline was designed to be self-contained and portable<\/strong>:<\/p>\n Practical implication:<\/strong> The same module runs in the production API, batch scripts, and evaluation notebooks\u2014without code changes.<\/p><\/blockquote>\n Enrichment improved search relevance measurably. The table below compares average cosine similarity scores between user queries and catalog results, with and without enrichment:<\/p>\n For assets that included reference images, the gains were even larger:<\/p>\n Reference images gave the model more visual context to work with, which translated directly into richer, more accurate metadata and stronger search alignment.<\/p>\n These techniques apply wherever an LLM produces structured output for production: medical record extraction, document classification, catalog enrichment, e-commerce product tagging.<\/p>\n While this pipeline focused on static images with text context<\/strong>, the same constrained-output-plus-validation pattern extends naturally to other modalities. You can process video assets by sampling keyframes, enriching each frame independently, and then merging results across the timeline. Audio and 3D assets follow the same principle\u2014extract structured metadata via a capable model, constrain it to known values, and validate before trusting.<\/p>\n The core principle: do not treat the LLM as a source of truth. Treat it as a first draft that must be validated, constrained, and evaluated against known-good data.<\/strong><\/p>\n A special thanks to Mark D’Souza<\/a>, Prabal Deb<\/a>, Preethi Natarajan<\/a>, Priya Bhimjyani<\/a> for their valuable input.<\/p>\nWhy Enrichment Instead of Just Embeddings?<\/h2>\n
\n\n
\n \nApproach<\/th>\n What it enables<\/th>\n What it cannot do<\/th>\n Output<\/th>\n What it means in practice<\/th>\n<\/tr>\n<\/thead>\n \n Embeddings<\/strong><\/td>\n “Find images similar to this one”<\/td>\n Tell you why<\/em> they are similar, or filter by specific attributes<\/td>\n Black-box vector<\/td>\n It can rank similar images, but it cannot tell you that an image contains a male protagonist in an indoor environment at night. There is no structured output to filter on.<\/td>\n<\/tr>\n \n Enrichment<\/strong><\/td>\n “Find all night scenes with a warrior holding a weapon”<\/td>\n Handle fuzzy, open-ended similarity queries<\/td>\n Structured, auditable fields<\/td>\n It produces tags, character roles, environment type, and prop categories. Each field is a testable claim that can be filtered, validated, and indexed.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n The Journey: Our Approach and Solution<\/h2>\n
What the Output Actually Does<\/h3>\n
{\r\n \"tags\": [\"palace\", \"night\", \"torch-lit\", \"warrior\", \"indoor\"],\r\n \"environment\": {\r\n \"type\": \"indoor\",\r\n \"time_of_day\": \"night\",\r\n \"lighting\": \"torch-lit\",\r\n \"realism\": \"cgi\"\r\n },\r\n \"characters\": [\r\n {\r\n \"name\": \"Warrior A\",\r\n \"role\": \"protagonist\",\r\n \"gender\": \"male\",\r\n \"species\": \"human\",\r\n \"position\": \"center\",\r\n \"action\": \"speaking\",\r\n \"visible_in_image\": true\r\n }\r\n ],\r\n \"props\": [\r\n {\r\n \"name\": \"ceremonial sword\",\r\n \"category\": \"weapon\",\r\n \"owner\": \"Warrior A\",\r\n \"visible_in_image\": true\r\n }\r\n ]\r\n}<\/code><\/pre>\nThe Flow<\/h3>\n
![\"Enrichment](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2026\/07\/ai-asset-enrichment-pipeline.webp\")
<\/p>\nThe Pipeline at a Glance<\/h3>\n
![\"Pipeline](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2026\/07\/pipeline-at-a-glance.webp\")
<\/p>\nStep 1: The Image-First Principle<\/h3>\n
visible_in_image=false<\/code>.<\/p><\/blockquote>\nStep 2: Preprocessing for Accuracy<\/h3>\n
\n
Step 3: Constrained Output\u2014The Key Design Decision<\/h3>\n
![\"Constrained](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2026\/07\/constrained-output-schema-scaled.webp\")
<\/p>\n\n\n
\n \nFree-text output<\/th>\n Constrained output<\/th>\n<\/tr>\n<\/thead>\n \n “The scene appears to take place during the evening hours, possibly dusk or early night”<\/em><\/td>\n time_of_day: \"night\"<\/code><\/td>\n<\/tr>\n\n Requires subjective judgment or another LLM to evaluate<\/td>\n Simple classification check: right or wrong<\/strong><\/td>\n<\/tr>\n \n Model can invent any value<\/td>\n Model must pick from a known set<\/td>\n<\/tr>\n \n Impossible to compute precision\/recall<\/td>\n Standard classification metrics apply directly<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n strict: true<\/code>). If the model produces an unlisted value, the validator coerces it to unknown<\/code> rather than propagating noise.<\/p>\nunknown<\/code> fallback is deliberate.<\/strong> It preserves the record without asserting false information. A system that says “I don’t know” is more trustworthy than one that guesses.<\/p><\/blockquote>\nStep 4: The 10-Stage Hallucination Defense<\/h3>\n
![\"10-Stage](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2026\/07\/10-stage-validation-scaled.webp\")
<\/p>\nvisible_in_image=false<\/code> and any unnamed prop. The model cannot override these filters.<\/p>\n\n
false<\/code> is removed. Period. This rule catches cases where context bias overrode visual evidence.<\/li>\n<\/ul>\nnull<\/code> (uncertain) are kept<\/strong>\u2014the pipeline prefers to retain uncertain data rather than silently discard it. Every removal is logged for auditability.<\/p>\nStep 5: Deterministic Description Generation<\/h3>\n
Evaluating the Pipeline: Ground Truth Template<\/h2>\n
\n\n
\n \nWhat to evaluate<\/th>\n How to measure it<\/th>\n Why it works<\/th>\n<\/tr>\n<\/thead>\n \n Environment type, time of day, lighting<\/td>\n Exact match<\/strong> against human annotation<\/td>\n Enum \u2192 simple right\/wrong check<\/td>\n<\/tr>\n \n Character presence and roles<\/td>\n Precision and recall<\/strong><\/td>\n Did the model find all characters? Did it add extras?<\/td>\n<\/tr>\n \n Prop identification and categories<\/td>\n Precision and recall<\/strong><\/td>\n Same as characters, for props<\/td>\n<\/tr>\n \n Tag quality<\/td>\n Overlap score<\/strong> (Jaccard similarity)<\/td>\n How much do generated tags overlap with human tags?<\/td>\n<\/tr>\n \n Visibility correctness<\/td>\n Binary accuracy<\/strong> on visible_in_image<\/code><\/td>\nThe hallucination-specific metric<\/td>\n<\/tr>\n \n Overall enrichment quality<\/td>\n Ground truth template comparison<\/strong><\/td>\n Compare full output against human-annotated expected output<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n unknown<\/code>), partially occluded props.<\/p>\nWhy a Plug-and-Play Module?<\/h2>\n
\n
VisionExtractionClient<\/code> with a single extract()<\/code> method. Swapping model providers is a configuration change, not a code change.<\/li>\nThe Destination: Outcomes and Learnings<\/h2>\n
Search Quality Impact<\/h3>\n
\n\n
\n \nComparison<\/th>\n Without Enrichment<\/th>\n With Enrichment<\/th>\n Gain<\/th>\n<\/tr>\n<\/thead>\n \n Query vs. Prompt (text only)<\/td>\n 0.50<\/td>\n 0.61<\/td>\n +22.5%<\/td>\n<\/tr>\n \n Query vs. Prompt + Image (combined)<\/td>\n 0.49<\/td>\n 0.59<\/td>\n +20.3%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n \n\n
\n \nComparison<\/th>\n Without Enrichment<\/th>\n With Enrichment<\/th>\n Gain<\/th>\n<\/tr>\n<\/thead>\n \n Query vs. Prompt (text only)<\/td>\n 0.46<\/td>\n 0.61<\/td>\n +32.7%<\/td>\n<\/tr>\n \n Query vs. Prompt + Image (combined)<\/td>\n 0.46<\/td>\n 0.59<\/td>\n +29.3%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Learnings<\/h3>\n
\n
unknown<\/code> rates revealed blind spots early.<\/strong> A high unknown<\/code> rate on a field signals the model consistently struggles\u2014that field may need few-shot examples or a specialized classifier.<\/li>\n<\/ol>\nThe Broader Pattern<\/h3>\n
Thanks<\/h2>\n