{"id":11655,"date":"2025-06-02T10:00:58","date_gmt":"2025-06-02T17:00:58","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/directx\/?p=11655"},"modified":"2025-09-16T11:30:24","modified_gmt":"2025-09-16T18:30:24","slug":"cooperative-vector","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/directx\/cooperative-vector\/","title":{"rendered":"D3D12 Cooperative Vector"},"content":{"rendered":"
\n

Welcome to the preview release for Cooperative Vector support in D3D12.\u00a0 This exposes powerful new hardware acceleration for vector and matrix operations, enabling developers to efficiently drive neural rendering techniques directly from individual shader threads in real-time graphics pipelines.<\/p>\n

In research and in industry, machine learning based approaches have made their way to mainstream, replacing\/augmenting traditional techniques. In graphics, neural network based rendering methods are gaining popularity over traditional methods of image reconstruction, texture compression, material shading etc. Simultaneously, the increasing use of GPUs for general purpose ML\/DL means that GPU vendors continue to add more specialized hardware in GPUs to accelerate neural network computations, like accelerating matrix operations.<\/p>\n

Parent blog for all other features in this release.<\/a><\/p>\n

\n

Motivation<\/h3>\n

Suppose we have a typical shader for lighting computation. This can be thousands of lines of computation, looping over light sources, evaluating complex materials. We want a way to replace these computations in individual shader threads with a neural network, with no other change to the rendering pipeline.\u00a0 The requested inference operation needs to be understood\u00a0 at a high-level by the driver so it can be mapped to dedicated hardware acceleration.<\/p>\n

\n

Feature Overview<\/h3>\n

Shader Model 6.9 adds a series of HLSL and DXIL features, building on each other, around vectors and matrix-vector operations available to shader threads.\u00a0 The result is that high level linear algebra operations can be directly consumed by drivers compiling shaders to GPUs with knowledge about how to take advantage of underlying hardware capability.<\/p>\n

Here is a summary of the features with links to the specs for each:<\/p>\n\n\n\n\n\n\n
HLSL Long Vector<\/a><\/strong><\/td>\nThe ability to load, store, and perform elementwise operations on HLSL vectors longer than four elements. In addition to the spec linked on the left, check out this overview: HLSL Native and Long Vectors Blog<\/a><\/td>\n<\/tr>\n
DXIL Vectors<\/a><\/strong><\/td>\nThe ability for vectors to appear in DXIL instead of being scalarized.<\/td>\n<\/tr>\n
HLSL Vector-Matrix Operations<\/a><\/strong><\/td>\nBuilds on Long Vectors above, allowing matrix-vector ops in HLSL to lower to the DXIL ops introduced in Cooperative Vector below.<\/td>\n<\/tr>\n
Cooperative Vector<\/strong><\/a><\/td>\nDXIL operations for vector-matrix operations that can be accelerated by the underlying hardware.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The “Cooperative” in Cooperative Vector refers to an implementation detail of the hardware acceleration, where individual vector-matrix multiply requests submitted by threads in a wave are combined into a matrix-matrix operation accelerated collectively for the wave.\u00a0 This name doesn’t appear in HLSL code itself, just vector types and operations like vector-matrix multiplication as shown in the examples below.<\/p>\n

Support for matrix-matrix operations is planned for the future.<\/p>\n

\n

Code Example<\/h3>\n
\/\/ Byte Address Buffers used to store vectors\/matrices\r\n\r\nByteAddressBuffer InVectors; \r\nRWByteAddressBuffer OutVectors;\r\nByteAddressBuffer InMatrices;\r\nRWByteAddressBuffer OutMatrices;\r\n\r\n\/\/ System header containing Cooperative Vector types, enums, and functions.\r\n#include <dx\/linalg.h>\r\n\r\n\/\/ Such elements are all under the linalg namespace.\r\nusing namespace dx::linalg;\r\n\r\n\/\/ Hand-wavey utility function to generate the input vector for Mul and MulAdd.\r\ntemplate<typename T, uint N> vector<T,N> GenerateVector(...);\r\n\r\n[numthreads(8,1,1)]\r\n[shader(\"compute\")]\r\nvoid main() \r\n{\r\n\r\n  \/\/ Matrix Vector Multiply Mul() Example\r\n\r\n  \/\/ Matrix and vector to be multiplied together\r\n  uint MatOffset = 0;\r\n  MatrixRef<DATA_TYPE_FLOAT32, 8, 6, MATRIX_LAYOUT_ROW_MAJOR> MulMatrix = {\r\n    InMatrices, MatOffset, \/*stride*\/6 * sizeof(float)};\r\n  MatOffset += 8 * 6 * sizeof(float);\r\n  \r\n  vector<float, 6> MulVector = GenerateVector<float, 6>(...);\r\n\r\n  vector<float, 8> MulRes =  Mul<float>(MulMatrix,\r\n                    MakeInterpretedVector<DATA_TYPE_FLOAT32>(MulVector));\r\n\r\n  \/\/ Matrix Vector Multiply and Add Bias Vector MulAdd() Example\r\n  MatrixRef<DATA_TYPE_FLOAT8_E4M3, 32, 4, MATRIX_LAYOUT_MUL_OPTIMAL> MulAddMatrix = {\r\n    InMatrices, MatOffset, \/*stride*\/0};\r\n  MatOffset += 32 * 4;\r\n  \r\n  half4 MulAddVector = GenerateVector<half, 4>(...);\r\n  VectorRef<DATA_TYPE_FLOAT8_E4M3> BiasVector = {InVectors, VecOffset};\r\n  VecOffset += 32;\r\n\r\n  vector<half, 32> MulAddRes =  MulAdd<half>(MulAddMatrix,\r\n                                             MakeInterpretedVector<DATA_TYPE_FLOAT8_E4M3>(MulAddVector),\r\n                                             BiasVector);\r\n\r\n  \/\/ Vector Vector Outer Product OuterProductAccumulate() Example\r\n  vector<uint8_t4_packed, 128> LeftVector = InVectors.Load< vector<uint8_t4_packed, 128> >(VecOffset);\r\n  VecOffset += sizeof(LeftVector);\r\n  vector<uint8_t4_packed, 64> RightVector = InVectors.Load< vector<uint8_t4_packed, 64> >(VecOffset);\r\n  VecOffset += sizeof(RightVector);\r\n  \r\n  \/\/ Storage for matrix produced by the outer product of above vectors.\r\n  RWMatrixRef<DATA_TYPE_UINT8, 128, 64, MATRIX_LAYOUT_OUTER_PRODUCT_OPTIMAL>\r\n  OuterMatrix = {OutMatrices, \/*offset*\/0, \/*stride*\/0};\r\n  \r\n  OuterProductAccumulate(LeftVector, RightVector, OuterMatrix);\r\n\r\n  \/\/ Vector Accumulating Addition VectorAccumulate() Example\r\n  vector<uint, 73> AccVector = InVectors.Load< vector<uint, 73> >(VecOffset);\r\n  VecOffset += sizeof(AccVector);\r\n  \r\n  VectorAccumulate(AccVector, OutVectors, 0 \/*offset*\/);\r\n  \r\n}<\/code><\/pre>\n
\n
Data preparation<\/h3>\n
There are a couple of D3D methods for converting weight and bias matrix data between formats:<\/p>\n
enum D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT {\r\n    D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT_ROW_MAJOR,\r\n    D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT_COLUMN_MAJOR,\r\n    D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT_MUL_OPTIMAL,\r\n    D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT_OUTER_PRODUCT_OPTIMAL\r\n}<\/code><\/pre>\n
For instance, D3D12_LINEAR_ALGEBRA_MATRIX_LAYOUT_MUL_OPTIMAL<\/code> is a device-specific layout for optimal use with the Cooperative Vector Matrix-Vector intrinsics such as the MulAdd<\/code> in the code example above.<\/p>\n
See ID3D12DevicePreview::GetLinearAlgebraMatrixConversionDestinationInfo()<\/code> and ID3D12CommandListPreview::ConvertLinearAlgebraMatrix()<\/code> in the Cooperative Vector spec here<\/a>.<\/p>\n
\n
Get running<\/h3>\n
Long Vector and Cooperative Vector are part of Shader Model 6.9, currently in preview. This requires:<\/p>\n