Additional Contributors: Justin Stoecker, Sheil Kumar

Overview

One of the biggest challenges in building AI/ML platform stacks for mobile, client, or server is creating a machine learning (ML) compiler that is both scalable yet performant. A common approach is to build specialized compilers for specific AI hardware, frameworks, or workload types. For example, independent hardware vendors (IHVs) provide their own frameworks to deploy models on their hardware, and PyTorch includes torch.compile as a framework-specific compiler. Other ML compiler efforts include MLIR (Multi-Level Intermediate Representation), which also provides compiler frameworks like Glow, EON, and others. Each approach serves particular focus areas, but few are designed to deliver high performance for ML workloads while remaining independent of hardware vendor and framework.

This post explores the challenges ML compiler developers face in pursuing strong performance without sacrificing scalability across diverse hardware architectures. It’s also a primer for BRK388 at Microsoft Ignite 2026, where we’ll go deeper on multi-level intermediate representations (IRs), including MLIR, and what they mean for local AI on Windows.

Introduction

Early graphics hardware began with fixed-function pipelines, which evolved into programmable shaders. The next leap introduced hybrid rasterization and ray tracing, and today these techniques are increasingly combined with neural graphics. Looking ahead, machine learning networks may play a larger role in rendering pipelines, changing how real-time graphics are built.

Deep neural networks are essentially graphs of mathematical operators, or layers. The natural instinct is to map this graph of operations to a graph of kernels, such as shaders. This approach feels intuitive and scalable but in practice, it’s slow in reaching time to market agility and extremely difficult to scale across evolving hardware architecture. The challenge lies in optimizing key operators, which is deceptively hard. The breadth of hardware makes handwritten solutions difficult to scale, and the rapid pace of AI innovation compounds this problem. Individual operator performance is often not the primary bottleneck; the greater complexity comes from achieving efficient execution across diverse hardware and evolving model architecture.

Optimizing Individual Layers

Matrix multiply-accumulate (MMA) is a core operator powering mainstream neural network architectures, including dense layers, MLPs, CNNs, and transformers. Dedicated MMA hardware has been present in GPU and NPU silicon for years and continues to expand. Naturally, this suggests we need a highly optimized MMA instruction at kernel level and modern hardware provides one. High-level abstractions provide portability but are often insufficient on their own to achieve hardware efficiency. Strong performance requires progressive lowering across multiple intermediate representation (IR) levels, enabling graph-level transformations, compute and memory fusion, hardware-aware tiling, data movement optimization, and execution scheduling that account for device-specific compute and memory hierarchies.

Device-Agnostic and Device-Aware Optimization

Device-agnostic rewrites standardize and optimize common patterns of operators and can be enabled at the framework level. Device-aware rewrites, such as decompositions and fusions, are driven by the device architecture and its instruction set architecture (ISA), meaning the set of operations the device can execute without an additional trip to memory. Pattern matching is one practical way to implement both, but it is not the only one. Infrastructure that supports both device-agnostic and device-aware graph optimization is an important early step for ML compilers that need to scale across hardware.

Fusion, Decomposition, and Control Access Layers

In languages with MMA support, the common expectation-versus-reality story is striking. You start with a model requiring X trillion operations (TOPS) and hardware claiming Y “AI TOPS” so you assume the model will run in X/Y seconds. In practice, you may often achieve only 5–10% of the expected throughput. The reason lies in the compute-to-memory access ratio required to maximize hardware efficiency hundreds of operations per byte of DRAM access for high-end GPUs. Algorithmically, it’s nearly impossible to reach this bar for most models if you process the network layer by layer. This is why operator fusion becomes critical, though the real questions remain: how much fusion is needed, and how much is even possible?

Fusion might sound simple, such as combining two adjacent operators, but in reality, it’s far more complex. It isn’t limited to element-wise operation or linear chains of operations. To truly optimize, fusions need to span heavyweight operations like matrix multiplication, convolution, reduction, and upsampling.

Fusion and decomposition don’t necessarily map to kernel boundaries. They are part of a broader question of how control and information are exchanged between the layers of a compiler.

Spanning Layers and Abstraction Levels

It’s best to think of fusion and decomposition as not purely about the scope of “kernels,” but as part of the broader problem of how control is shared across layers. In a cross-device ML compiler, sometimes you want to provide a backend with a large graph to reason about end-to-end execution. Sometimes you want to carve out subgraphs where hardware can do something special. And sometimes you need tile or kernel level control to squeeze out additional performance. In practice, this means any serious stack ends up spanning multiple IRs and abstraction levels.

The tricky part isn’t just having multiple layers – it’s that responsibility gets split across them. This is where performance becomes surprisingly fragile. If an upper layer simply punts decisions downward (“the lower layer will optimize it”), outcomes become hard to predict. It’s not only about whether individual operators and data types are supported, but how efficiently patterns of connected operators are handled. Without proper understanding, the upper layer can’t reliably tell which parts of the graph will stay on fast paths versus triggering slow paths, fallbacks, or late reshaping of the computation.

When those surprises happen late, they’re expensive. You may have to backtrack and redo earlier choices specially for each hardware target, such as hand-editing models, changing how a compiler partitions the graph, how it orders transformations, or how it introduces tiling and layout, just to avoid unintended cliffs. This “you only find out at the end” dynamic is one of the core challenges in building scalable, high-performance ML compilers across diverse hardware.

Approaching this problem purely as choice of static primitive granularity combined with greedy partitioning is often not enough. What’s needed is a richer cooperative interface for sharing graph-level visibility and control across component layers.

Scheduling and Memory Management

Optimizing machine learning workloads requires deep understanding of associated hardware architecture to be able to utilize hardware resources efficiently. This includes interconnected decisions around tensor layout, splitting and streaming algorithms, cache memory usage, data formats, tiling, and compute stencils. Equally critical is scheduling compute and memory tasks in a pipelined fashion, carefully balancing space, and time constraints to achieve optimal performance. As these software algorithms evolve, it’s essential to maintain a dual focus: scalability across diverse platform architectures and maximizing device specific performance gains.

These details are also highly coupled with hardware architecture details such as MMA sizes, on-chip memory capacity, register file limits, and data sharing and synchronization across threads, waves, thread groups, and clusters. For example, in client-side inference scenario, a real-time models might run with batch size 1 and strict latency requirements of around 2ms, making it impractical to shuttle large amounts of data through DRAM. In server-side inference, constraints can shift with high-bandwidth memory (HBM) and larger batch sizes.

Equally important is scheduling compute and memory tasks in a pipelined fashion, carefully balancing space, and time constraints to achieve strong performance. As these software algorithms evolve, it’s important to maintain a dual focus: scalability across diverse platform architectures and maximizing device-specific performance gains.

Ultimately, executing complex networks efficiently is as challenging as building a high-performance rendering engine; it’s far more than writing a single compute shader.

Conclusion

The industry faces a broad challenge that calls for a unified platform-level solution, and it can be approached at two layers. At the higher level, solutions focused on frontend frameworks offer ease of use and often deliver good performance, but can face challenges with cross-device consistency, unified tooling, potentially unbounded compilation times, limited low-level control, and interoperability with graphics. At the lower level, solutions for individual operators can operate well with graphics and low-level engines and are generally independent of a specific hardware vendor. However, they require more effort to use and raise concerns around scalability and performance compared to vendor-specific solutions.

The solution likely lies between these levels. It calls for a structured process for defining use cases, understanding technical requirements, and making informed architectural decisions. We see an opportunity for Windows ML and DirectX to help address this challenge across a range of hardware architectures.As we look ahead, we will share more about how we’re approaching these challenges. What if developers could combine the scalability of a unified ML compiler architecture with the performance of hardware-specific optimization? We’re exploring how DirectX Compute Graph Compiler (DXCGC), Windows ML, and a multi-level IR architecture could help developers bring AI workloads to diverse hardware with greater consistency and portability.

Stay tuned for Microsoft Ignite 2026 next week, where we will share more about our direction for local AI development on Windows in BRK388 – be sure to check it out!