TL;DR

Running a Retrieval-Augmented Generation (RAG) system efficiently on Windows, without internet access, and with strict performance constraints requires selecting the right inference engine. After evaluating ONNX Runtime GenAI, LlamaCPP, Hugging Face Optimum, and Triton, we found that ONNX Runtime GenAI outperformed other solutions in token throughput, latency, and wall-clock efficiency for our scenario, making it the best choice for our deployment. In this post, we break down the evaluation process, performance results, and integration details.

Introduction

Running SLM inference on Windows is often challenging due to dependency issues, lack of native support for some frameworks, and performance bottlenecks. Our goal was to build a RAG system with LangChain, ONNX Runtime GenAI, and Guardrails that can be deployed on Windows environment, ensuring:

• All computations happen locally (no cloud dependencies)
• Without internet connection
• Inference is completed under 5 seconds
• Efficient model deployment on a Windows on-prem environment

Given these constraints, we evaluated four inference engines:

• ONNX Runtime GenAI (docs)
• LlamaCPP (GitHub)
• Hugging Face Optimum (GitHub)
• Triton Inference Server (NVIDIA Triton)

Problem Statement

Given the constraints of a Windows-based RAG system, we needed to determine which inference engine would allow us to:

• Run LLM inference with minimal latency (<5s total)
• Ensure maximum GPU utilization
• Optimize token generation throughput
• Remain compatible with LangChain integration

We used Microsoft’s Phi-3 Mini ONNX model (Hugging Face link) for testing, deployed on an Azure ND96amsr_A100_v4 instance (1x A100 80GB GPU).

For benchmarking, we used fixed generation length 256 tokens, with 5 warmup run and 10 repetition.

Decision: Why ONNX Runtime GenAI?

After benchmarking different inference solutions, ONNX Runtime GenAI was selected because:

• Performance: It outperformed all other options in token throughput, latency, and GPU utilization.
• Compatibility: Unlike Optimum, it allows full optimization for Phi-3 ONNX models.
• Windows Support: Unlike LlamaCPP, it doesn’t require extensive custom builds.
• Deployment Flexibility: We cannot use Triton due to on-prem restrictions (no Docker).

Performance Benchmarking

We evaluated the solutions based on:

• Token throughput (Tokens per second)
• Token generation latency
• Wall-clock latency
• Wall-clock throughput
• GPU utilization (Target: ~99%)

For benchmark, we have used script from OnnxruntimeGenai package, here is the link.

Average Token Generation Throughput (tokens per second, tps)

Average Wall-Clock Throughput (tps)

Average Wall-Clock (s)

Optimum vs ONNX Runtime GenAI

Optimum’s graph optimization is limited, and it doesn’t support Phi-3. Using ONNX Runtime GenAI’s builder, we could achieve significantly better throughput and lower latency.

Wall-clock Latency Optimum vs ONNX Runtime GenAI

Integration with LangChain

To integrate ONNX Runtime GenAI with LangChain, we extended BaseLLM and BaseChatModel to support ONNX inference:

def _generate(self, prompts, stop = None, run_manager = None, **kwargs):

        from onnxruntime_genai import GeneratorParams,Generator
        text_generations: list[str] = []
        answer:str=""

        # Encode prompts
        input_token = self.tokenizer.encode_batch(prompts)

        model_params = self._default_params
        model_params.update(self.model_kwargs)

        # Build generator params
        params = GeneratorParams(self.model)
        params.set_search_options(**model_params)
        generator = Generator(self.model, params)

        # Append input token
        generator.append_tokens(input_token)
        while not generator.is_done():
            generator.generate_next_token()
            new_token = generator.get_next_tokens()[0]

            answer+=self.tokenizer.decode(new_token)

            print(self.tokenizer_stream.decode(new_token), end='', flush=True)
        text_generations.append(answer)

        del generator

        return LLMResult(generations=[[Generation(text=text) for text in text_generations]])

We also created an ONNX conversion job using onnxruntime-genai.builder, This ensures optimized models for inference.

onnxruntime-genai.builder --model phi-3-mini --output ./phi3_optimized.onnx

Conclusion

Choosing the right inference engine matters for on-prem RAG systems, especially on Windows. ONNX Runtime GenAI provided:

• Best performance (higher throughput, lower latency)
• Easy integration (LangChain & onnxruntime-genai.builder)
• Optimized models for local inference

If you’re working on Windows-based AI deployments, ONNX Runtime GenAI is the way to go.

References

• ONNX Runtime GenAI documentation
• LlamaCPP
• Hugging Face Optimum
• NVIDIA Triton
• Phi3-mini-4k-instruct-Onnx Hugging Face