Introduction: The Problem

Imagine a catalog of 50,000 visual assets—characters, environments, props—and no way to search them except scrolling. Someone asks: “Find me all the night scenes with a warrior holding a weapon.” Without structured metadata, that search is impossible.

Manual tagging does not scale. We needed a way to automatically extract rich, structured metadata from images—and critically, to trust the output enough to put it in a production search index.

The core challenge: multimodal LLMs are excellent at describing images, but they hallucinate. They confidently report characters not in the scene and invent props that do not exist. For a search catalog, a hallucinated tag is worse than a missing one—it erodes trust in the system.

The question was not “can an LLM extract metadata from images?”—it was “can we make the output reliable enough to trust at scale?”

Why Enrichment Instead of Just Embeddings?

Enrichment and embeddings are complementary. Enrichment enables faceted search and filtering. Embeddings enable semantic similarity. Use both.

The Journey: Our Approach and Solution

What the Output Actually Does

Before diving into how it works, here is what the enrichment pipeline produces. Given a scene image, the pipeline outputs structured JSON:

{
  "tags": ["palace", "night", "torch-lit", "warrior", "indoor"],
  "environment": {
    "type": "indoor",
    "time_of_day": "night",
    "lighting": "torch-lit",
    "realism": "cgi"
  },
  "characters": [
    {
      "name": "Warrior A",
      "role": "protagonist",
      "gender": "male",
      "species": "human",
      "position": "center",
      "action": "speaking",
      "visible_in_image": true
    }
  ],
  "props": [
    {
      "name": "ceremonial sword",
      "category": "weapon",
      "owner": "Warrior A",
      "visible_in_image": true
    }
  ]
}

Every value in that JSON comes from a constrained set of options—not free text. That design decision makes the entire pipeline evaluable, and it ties the rest of this post together.

The Flow

The following diagram shows the complete flow—from input image through the pipeline to validated metadata:

The Pipeline at a Glance

Step 1: The Image-First Principle

When a production team provides context—“this scene has characters A, B, and C”—the model is tempted to report all three even if only one appears. Context biases the model toward expected content rather than observed content.

Rule: The final image is ground truth. Reference images and text help the model identify and name elements—but only elements it can actually see. Everything else must be marked visible_in_image=false.

Think of it like a witness in court—background knowledge helps identify people, but testimony must cover only what was personally observed.

Step 2: Preprocessing for Accuracy

• Resize to 1,024px (preserving aspect ratio)—balances fidelity against token cost
• LANCZOS resampling—preserves fine details like text on props and facial features
• Message ordering—reference images first, final asset last, then the prompt

Step 3: Constrained Output—The Key Design Decision

Instead of asking the model “describe the environment” (free text, impossible to evaluate), the pipeline asks “classify the environment into one of these categories”:

Why this matters:

The model’s output is enforced via JSON schema mode (strict: true). If the model produces an unlisted value, the validator coerces it to unknown rather than propagating noise.

The unknown fallback is deliberate. It preserves the record without asserting false information. A system that says “I don’t know” is more trustworthy than one that guesses.

Step 4: The 10-Stage Hallucination Defense

Raw model output—even with structured JSON enforced—is never trusted directly. A 10-stage validation pipeline acts as an independent safety net:

Stages 6, 7, and 8 are the hallucination guards—they unconditionally remove any character or prop marked visible_in_image=false and any unnamed prop. The model cannot override these filters.

• Prompt layer (soft): The model is instructed to be honest about visibility—this handles most cases correctly.
• Validator layer (hard): Anything marked false is removed. Period. This rule catches cases where context bias overrode visual evidence.

Items marked null (uncertain) are kept—the pipeline prefers to retain uncertain data rather than silently discard it. Every removal is logged for auditability.

Analogy: The prompt layer is like training an employee to be accurate. The validator layer is like having a supervisor check every report. You need both.

Step 5: Deterministic Description Generation

The human-readable descriptions (“A torch-lit indoor palace at night with a warrior protagonist”) are not generated by the LLM. They are composed deterministically from the validated structured fields—so descriptions always match the structured data. No contradictions, no drift.

Evaluating the Pipeline: Ground Truth Template

Because every output field is constrained to a finite set of values, evaluation becomes a classification problem—not a subjective comparison. We proposed a ground truth evaluation template:

The template also captures edge cases: images with no characters (expect an empty list), ambiguous lighting (expect unknown), partially occluded props.

Key insight: Constrained output makes evaluation tractable. Enum-constrained evaluation is a standard classification problem.

Why a Plug-and-Play Module?

The enrichment pipeline was designed to be self-contained and portable:

• One abstract interface—The pipeline depends on a VisionExtractionClient with a single extract() method. Swapping model providers is a configuration change, not a code change.
• Isolated configuration—All enrichment settings live in a dedicated config. The module has zero knowledge of other modules.
• Self-contained contracts—The module defines its own input/output schemas. Any system that sends a request and consumes the response can use it.
• Optional by design—Enable or disable enrichment at runtime. No errors when disabled.

Practical implication: The same module runs in the production API, batch scripts, and evaluation notebooks—without code changes.

The Destination: Outcomes and Learnings

Search Quality Impact

Enrichment improved search relevance measurably. The table below compares average cosine similarity scores between user queries and catalog results, with and without enrichment:

For assets that included reference images, the gains were even larger:

Reference images gave the model more visual context to work with, which translated directly into richer, more accurate metadata and stronger search alignment.

Learnings

1. Constrained output was the highest-leverage decision. It simultaneously minimized hallucination (model cannot invent values), enabled evaluation (classification metrics, not subjective judgment), and improved downstream search quality (clean, consistent facets).
2. Two layers of hallucination defense caught different failure modes. The prompt layer handled most cases; the validator caught the remaining ~5-10% where context bias overrode visual evidence. Neither alone was sufficient.
3. Deterministic descriptions eliminated a hallucination surface by not asking the model for free text.
4. The plug-and-play design paid off immediately. The same module ran in the production API, batch backfill scripts, and evaluation notebooks—without modification.
5. Tracking unknown rates revealed blind spots early. A high unknown rate on a field signals the model consistently struggles—that field may need few-shot examples or a specialized classifier.

The Broader Pattern

These techniques apply wherever an LLM produces structured output for production: medical record extraction, document classification, catalog enrichment, e-commerce product tagging.

While this pipeline focused on static images with text context, the same constrained-output-plus-validation pattern extends naturally to other modalities. You can process video assets by sampling keyframes, enriching each frame independently, and then merging results across the timeline. Audio and 3D assets follow the same principle—extract structured metadata via a capable model, constrain it to known values, and validate before trusting.

The core principle: do not treat the LLM as a source of truth. Treat it as a first draft that must be validated, constrained, and evaluated against known-good data.

Thanks

A special thanks to Mark D’Souza, Prabal Deb, Preethi Natarajan, Priya Bhimjyani for their valuable input.

The feature image was generated using Copilot. Terms can be found at Microsoft Copilot Terms of Use.