{"id":16728,"date":"2026-09-14T00:00:00","date_gmt":"2026-09-14T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ise\/?p=16728"},"modified":"2026-09-13T05:50:29","modified_gmt":"2026-09-13T12:50:29","slug":"japanese-llm-evaluation-pipeline-hackathon","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/japanese-llm-evaluation-pipeline-hackathon\/","title":{"rendered":"Building a Japanese LLM Evaluation Pipeline: Lessons from a Two-Day Hackathon"},"content":{"rendered":"

Introduction<\/h2>\n

A: \u304b\u3057\u3053\u307e\u308a\u307e\u3057\u305f\u3002\u3054\u6ce8\u6587\u756a\u53f7\u3092\u304a\u6559\u3048\u3044\u305f\u3060\u3051\u308c\u3070\u3001\u914d\u9001\u72b6\u6cc1\u3092\u304a\u8abf\u3079\u3044\u305f\u3057\u307e\u3059\u3002<\/p>\n

B: \u3054\u4e0d\u5b89\u3067\u3059\u3088\u306d\u3002\u3054\u6ce8\u6587\u756a\u53f7\u3092\u304a\u6559\u3048\u3044\u305f\u3060\u3051\u308c\u3070\u3001\u3059\u3050\u306b\u304a\u8abf\u3079\u3044\u305f\u3057\u307e\u3059\u3002<\/p>\n

Which sentence do you think is more appropriate as a response from call center AI? As voice AI has evolved, expectations in call centers and customer service have shifted from traditional IVR (Interactive Voice Response) systems toward more natural and flexible conversational experiences. Services like wevnal’s “BOTCHAN AI Call” require more than simple FAQ responses\u2014they demand human operator-level dialogue quality.<\/p>\n

When selecting an LLM for customer-facing conversational features in Japanese, leaderboards often fail to serve as reliable references for specific scenarios. MT-Bench, MMLU, and similar benchmarks are English-centric, single-turn, and accuracy-focused. Conversational nuance in Japanese \u2014 appropriate keigo, emotional reading of a frustrated customer, persona consistency across a multi-turn exchange \u2014 rarely shows up in the numbers. As a result, teams end up trial-running models against in-house prompts, by hand, every time.<\/p>\n

In April 2026, we joined wevnal for a two-day hackathon to close that gap: build a benchmark and evaluation harness that is multi-provider, multi-turn, and oriented around facets<\/em> of conversational quality, with the vision to contribute back to the Japanese AI community. This post walks through how it is built, why it is built that way, and what we learned along the way.<\/p>\n

The Problem We Needed to Solve<\/h2>\n

Subjectivity without standards<\/strong>: No shared rubrics exist for “Naturalness” in Japanese conversation, for what makes a response feel contextually appropriate or emotionally attuned to the user’s tone.<\/p>\n

Fragmented evaluation<\/strong>: Existing efforts covered isolated pieces\u2014RAG components here, single-turn accuracy there\u2014but no end-to-end pipeline tied them together.<\/p>\n

Data scarcity<\/strong>: High-quality Japanese conversational datasets are rare. Without ground truth or established baselines, it was hard to tell whether model changes were real improvements.<\/p>\n

Manual, slow iteration<\/strong>: Evaluation was largely manual and disconnected from CI\/CD\u2014too slow to systematically improve quality at the pace the product needed.<\/p>\n

Closing this gap required an automated, standardized, production-ready pipeline that could scale across hundreds of different models.<\/p>\n

A note on what this benchmark is for<\/h2>\n

We treat the benchmark as an initial filter<\/strong>, not a final verdict. Its job is to narrow 50 candidate models down to the 3\u20135 worth deeper, per-product evaluation \u2014 which engineering teams should still own. That framing drives the design choices below: optimize for breadth and reproducibility<\/strong> over depth, and keep metrics swappable as the field’s notion of “good Japanese conversation” evolves.<\/p>\n

On inspiration and licensing<\/h2>\n

\"Architecture<\/p>\n

The first architectural decision was to keep two things strictly separate, both in code and in mental model:<\/p>\n