{"id":16174,"date":"2025-05-02T00:00:00","date_gmt":"2025-05-02T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ise\/?p=16174"},"modified":"2025-05-02T04:07:06","modified_gmt":"2025-05-02T11:07:06","slug":"running-rag-onnxruntime-genai","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/running-rag-onnxruntime-genai\/","title":{"rendered":"Running RAG with ONNX Runtime GenAI for On-Prem Windows"},"content":{"rendered":"<h1>TL;DR<\/h1>\n<p>Running a <strong>Retrieval-Augmented Generation (RAG) system<\/strong> efficiently on <strong>Windows<\/strong>, without internet access, and with strict performance constraints requires selecting the right inference engine. After evaluating <strong>ONNX Runtime GenAI, LlamaCPP, Hugging Face Optimum, and Triton<\/strong>, we found that <strong>ONNX Runtime GenAI<\/strong> outperformed other solutions in <strong>token throughput, latency, and wall-clock efficiency<\/strong> for our scenario, making it the best choice for our deployment. In this post, we break down the evaluation process, performance results, and integration details.<\/p>\n<h2>Introduction<\/h2>\n<p>Running <strong>SLM inference<\/strong> on Windows is often challenging due to <strong>dependency issues, lack of native support for some frameworks, and performance bottlenecks<\/strong>. Our goal was to build a <strong>RAG system with LangChain, ONNX Runtime GenAI, and Guardrails<\/strong> that can be deployed on Windows environment, ensuring:<\/p>\n<ul>\n<li><strong>All computations happen locally<\/strong> (no cloud dependencies)<\/li>\n<li><strong>Without internet connection<\/strong><\/li>\n<li><strong>Inference is completed under 5 seconds<\/strong><\/li>\n<li><strong>Efficient model deployment on a Windows on-prem environment<\/strong><\/li>\n<\/ul>\n<p>Given these constraints, we evaluated <strong>four inference engines<\/strong>:<\/p>\n<ul>\n<li><strong>ONNX Runtime GenAI<\/strong> (<a href=\"https:\/\/onnxruntime.ai\/docs\/genai\/\">docs<\/a>)<\/li>\n<li><strong>LlamaCPP<\/strong> (<a href=\"https:\/\/github.com\/ggerganov\/llama.cpp\">GitHub<\/a>)<\/li>\n<li><strong>Hugging Face Optimum<\/strong> (<a href=\"https:\/\/github.com\/huggingface\/optimum\">GitHub<\/a>)<\/li>\n<li><strong>Triton Inference Server<\/strong> (<a href=\"https:\/\/developer.nvidia.com\/nvidia-triton-inference-server\">NVIDIA Triton<\/a>)<\/li>\n<\/ul>\n<h2>Problem Statement<\/h2>\n<p>Given the constraints of a <strong>Windows-based RAG system<\/strong>, we needed to determine which inference engine would allow us to:<\/p>\n<ul>\n<li><strong>Run LLM inference with minimal latency (&lt;5s total)<\/strong><\/li>\n<li><strong>Ensure maximum GPU utilization<\/strong><\/li>\n<li><strong>Optimize token generation throughput<\/strong><\/li>\n<li><strong>Remain compatible with LangChain integration<\/strong><\/li>\n<\/ul>\n<p>We used <strong>Microsoft\u2019s Phi-3 Mini ONNX model<\/strong> (<a href=\"https:\/\/huggingface.co\/microsoft\/Phi-3-mini-4k-instruct-onnx\">Hugging Face link<\/a>) for testing, deployed on an <strong>Azure ND96amsr_A100_v4 instance<\/strong> (1x A100 80GB GPU).<\/p>\n<p>For benchmarking, we used fixed generation length <strong>256 tokens<\/strong>, with <strong>5<\/strong> warmup run and <strong>10<\/strong> repetition.<\/p>\n<h2>Decision: Why ONNX Runtime GenAI?<\/h2>\n<p>After benchmarking different inference solutions, <strong>ONNX Runtime GenAI<\/strong> was selected because:<\/p>\n<ul>\n<li><strong>Performance:<\/strong> It outperformed all other options in <strong>token throughput, latency, and GPU utilization<\/strong>.<\/li>\n<li><strong>Compatibility:<\/strong> Unlike <strong>Optimum<\/strong>, it allows full optimization for <strong>Phi-3 ONNX models<\/strong>.<\/li>\n<li><strong>Windows Support:<\/strong> Unlike <strong>LlamaCPP<\/strong>, it doesn\u2019t require extensive custom builds.<\/li>\n<li><strong>Deployment Flexibility:<\/strong> We cannot use <strong>Triton<\/strong> due to on-prem restrictions (no Docker).<\/li>\n<\/ul>\n<h3>Performance Benchmarking<\/h3>\n<p>We evaluated the solutions based on:<\/p>\n<ul>\n<li><strong>Token throughput<\/strong> (Tokens per second)<\/li>\n<li><strong>Token generation latency<\/strong><\/li>\n<li><strong>Wall-clock latency<\/strong><\/li>\n<li><strong>Wall-clock throughput<\/strong><\/li>\n<li><strong>GPU utilization<\/strong> (Target: ~99%)<\/li>\n<\/ul>\n<p>For benchmark, we have used script from <code>OnnxruntimeGenai<\/code> package, here is the <a href=\"https:\/\/github.com\/microsoft\/onnxruntime-genai\/tree\/main\/benchmark\/python\">link<\/a>.<\/p>\n<h3>Average Token Generation Throughput (tokens per second, tps)<\/h3>\n<table>\n<thead>\n<tr>\n<th>Batch Size<\/th>\n<th>Prompt Length<\/th>\n<th>Onnxruntime-genai<\/th>\n<th>LlamaCPP<\/th>\n<th>HF Optimum<\/th>\n<th>Speed Up ORT\/LlamaCPP Ratio<\/th>\n<th>Speed Up ORT\/Optimum Ratio<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>1<\/td>\n<td>16<\/td>\n<td>137.59<\/td>\n<td>109.47<\/td>\n<td>108.345<\/td>\n<td>1.26<\/td>\n<td>1.27<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>64<\/td>\n<td>136.82<\/td>\n<td>110.26<\/td>\n<td>107.135<\/td>\n<td>1.24<\/td>\n<td>1.28<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>256<\/td>\n<td>134.45<\/td>\n<td>109.42<\/td>\n<td>105.755<\/td>\n<td>1.23<\/td>\n<td>1.36<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1024<\/td>\n<td>127.34<\/td>\n<td>105.60<\/td>\n<td>102.114<\/td>\n<td>1.21<\/td>\n<td>1.50<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>2048<\/td>\n<td>122.62<\/td>\n<td>102.00<\/td>\n<td>99.345<\/td>\n<td>1.20<\/td>\n<td>1.59<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1.png\"><img decoding=\"async\" class=\"alignnone wp-image-16189 size-full\" src=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1.png\" alt=\"Average Token Generation Throughput\" width=\"1663\" height=\"999\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1.png 1663w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1-300x180.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1-1024x615.png 1024w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1-768x461.png 768w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/token_generation_linux_1-1536x923.png 1536w\" sizes=\"(max-width: 1663px) 100vw, 1663px\" \/><\/a><\/p>\n<p>&nbsp;<\/p>\n<h3><strong>Average Wall-Clock Throughput (tps)<\/strong><\/h3>\n<table>\n<thead>\n<tr>\n<th>Prompt Length<\/th>\n<th>Windows<\/th>\n<th>Linux<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>16<\/td>\n<td>142.18<\/td>\n<td>129.52<\/td>\n<\/tr>\n<tr>\n<td>64<\/td>\n<td>166.20<\/td>\n<td>154.18<\/td>\n<\/tr>\n<tr>\n<td>256<\/td>\n<td>259.48<\/td>\n<td>235.68<\/td>\n<\/tr>\n<tr>\n<td>1024<\/td>\n<td>585.68<\/td>\n<td>545.91<\/td>\n<\/tr>\n<tr>\n<td>2048<\/td>\n<td>932.03<\/td>\n<td>892.12<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/wall_clock_vs_phi3_linux.png\" alt=\"Average Wall-Clock Throughput\" \/><\/p>\n<h3><strong>Average Wall-Clock (s)<\/strong><\/h3>\n<table>\n<thead>\n<tr>\n<th>Prompt Length<\/th>\n<th>Windows<\/th>\n<th>Linux<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>16<\/td>\n<td>1.913120966<\/td>\n<td>2.1<\/td>\n<\/tr>\n<tr>\n<td>64<\/td>\n<td>1.925366235<\/td>\n<td>2.08<\/td>\n<\/tr>\n<tr>\n<td>256<\/td>\n<td>1.97314712<\/td>\n<td>2.17<\/td>\n<\/tr>\n<tr>\n<td>1024<\/td>\n<td>2.185485754<\/td>\n<td>2.34<\/td>\n<\/tr>\n<tr>\n<td>2048<\/td>\n<td>2.472010641<\/td>\n<td>2.58<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/wall_clock_latency_vs_phi3_linux.png\" alt=\"Average Wall-Clock\" \/><\/p>\n<h3>Optimum vs ONNX Runtime GenAI<\/h3>\n<p>Optimum\u2019s <strong>graph optimization<\/strong> is limited, and it doesn\u2019t support <strong>Phi-3<\/strong>. Using <strong>ONNX Runtime GenAI&#8217;s builder<\/strong>, we could achieve <strong>significantly better throughput and lower latency<\/strong>.<\/p>\n<h3>Wall-clock Latency Optimum vs ONNX Runtime GenAI<\/h3>\n<table>\n<thead>\n<tr>\n<th>Prompt Length<\/th>\n<th>ONNX Runtime GenAI (s)<\/th>\n<th>Optimum (s)<\/th>\n<th>Llamacpp (s)<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>16<\/td>\n<td>2.491<\/td>\n<td>3.526<\/td>\n<td>2.51<\/td>\n<\/tr>\n<tr>\n<td>64<\/td>\n<td>2.502<\/td>\n<td>3.545<\/td>\n<td>2.52<\/td>\n<\/tr>\n<tr>\n<td>256<\/td>\n<td>2.571<\/td>\n<td>3.772<\/td>\n<td>2.71<\/td>\n<\/tr>\n<tr>\n<td>1024<\/td>\n<td>2.790<\/td>\n<td>4.049<\/td>\n<td>3.12<\/td>\n<\/tr>\n<tr>\n<td>2048<\/td>\n<td>3.073<\/td>\n<td>4.279<\/td>\n<td>3.46<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/05\/wall_clock_latency_optimum_vs_genai.png\" alt=\"Wall-Clock Comparison\" \/><\/p>\n<h3>Integration with LangChain<\/h3>\n<p>To integrate ONNX Runtime GenAI with <strong>LangChain<\/strong>, we extended <code>BaseLLM<\/code> and <code>BaseChatModel<\/code> to support <strong>ONNX inference<\/strong>:<\/p>\n<pre><code class=\"language-python\">def _generate(self, prompts, stop = None, run_manager = None, **kwargs):\r\n\r\n        from onnxruntime_genai import GeneratorParams,Generator\r\n        text_generations: list[str] = []\r\n        answer:str=\"\"\r\n\r\n        # Encode prompts\r\n        input_token = self.tokenizer.encode_batch(prompts)\r\n\r\n        model_params = self._default_params\r\n        model_params.update(self.model_kwargs)\r\n\r\n        # Build generator params\r\n        params = GeneratorParams(self.model)\r\n        params.set_search_options(**model_params)\r\n        generator = Generator(self.model, params)\r\n\r\n        # Append input token\r\n        generator.append_tokens(input_token)\r\n        while not generator.is_done():\r\n            generator.generate_next_token()\r\n            new_token = generator.get_next_tokens()[0]\r\n\r\n            answer+=self.tokenizer.decode(new_token)\r\n\r\n            print(self.tokenizer_stream.decode(new_token), end='', flush=True)\r\n        text_generations.append(answer)\r\n\r\n        del generator\r\n\r\n        return LLMResult(generations=[[Generation(text=text) for text in text_generations]])<\/code><\/pre>\n<p>We also created an <strong>ONNX conversion job<\/strong> using <code>onnxruntime-genai.builder<\/code>, This ensures <strong>optimized models<\/strong> for inference.<\/p>\n<pre><code class=\"language-bash\">onnxruntime-genai.builder --model phi-3-mini --output .\/phi3_optimized.onnx<\/code><\/pre>\n<h2>Conclusion<\/h2>\n<p>Choosing the right inference engine <strong>matters<\/strong> for on-prem RAG systems, especially on <strong>Windows<\/strong>. ONNX Runtime GenAI provided:<\/p>\n<ul>\n<li><strong>Best performance<\/strong> (higher throughput, lower latency)<\/li>\n<li><strong>Easy integration<\/strong> (LangChain &amp; onnxruntime-genai.builder)<\/li>\n<li><strong>Optimized models<\/strong> for local inference<\/li>\n<\/ul>\n<p>If you&#8217;re working on Windows-based AI deployments, <strong>ONNX Runtime GenAI is the way to go<\/strong>.<\/p>\n<h2>References<\/h2>\n<ul>\n<li><a href=\"https:\/\/onnxruntime.ai\/docs\/genai\/\">ONNX Runtime GenAI documentation<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggerganov\/llama.cpp\">LlamaCPP<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/huggingface\/optimum\">Hugging Face Optimum<\/a><\/li>\n<li><a href=\"https:\/\/developer.nvidia.com\/nvidia-triton-inference-server\">NVIDIA Triton<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/microsoft\/Phi-3-mini-4k-instruct-onnx\">Phi3-mini-4k-instruct-Onnx Hugging Face<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Exploring how to efficiently run a RAG pipeline with structured language models (SLMs) and guardrails on Windows, achieving inference under 5 seconds with ONNX Runtime GenAI.<\/p>\n","protected":false},"author":172176,"featured_media":16189,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,3451],"tags":[3595,238,3594,3593],"class_list":["post-16174","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse","category-ise","tag-inferencing","tag-machine-learning","tag-onnxruntime","tag-slm"],"acf":[],"blog_post_summary":"<p>Exploring how to efficiently run a RAG pipeline with structured language models (SLMs) and guardrails on Windows, achieving inference under 5 seconds with ONNX Runtime GenAI.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16174","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/172176"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=16174"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/16174\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/16189"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=16174"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=16174"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=16174"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}