.NET has a long history of providing access to additional hardware functionality via APIs that are intrinsically understood by the JIT compiler. This started on .NET Framework back in 2014 and expanded with the introduction of .NET Core 3.0 in 2019. Since then, the runtime has iteratively provided more APIs and taken better advantage of this in each release.<\/p>\n
As a brief overview:<\/p>\n
System.Numerics<\/code> namespace\n\n- Introduces
Vector<T><\/code><\/li>\n- Introduces
Vector2<\/code>, Vector3<\/code>, Vector4<\/code>, Matrix4x4<\/code>, Quaternion<\/code>, and Plane<\/code><\/li>\n- 64-bit only<\/li>\n
- See also: https:\/\/devblogs.microsoft.com\/dotnet\/the-jit-finally-proposed-jit-and-simd-are-getting-married\/<\/li>\n<\/ul>\n<\/li>\n
- 2019 – .NET Core 3.0 – First APIs exposed in the
System.Runtime.Intrinsics<\/code> namespace\n\n- Introduces
Vector128<T><\/code> and Vector256<T><\/code><\/li>\n- Introduces
Sse<\/code>, Sse2<\/code>, Sse3<\/code>, Ssse3<\/code>, Sse41<\/code>, Sse42<\/code>, Avx<\/code>, Avx2<\/code>, Fma<\/code>, Bmi1<\/code>, Bmi2<\/code>, Lzcnt<\/code>, Popcnt<\/code>, Aes<\/code>, and Pclmul<\/code> for x86<\/code>\/x64<\/code><\/li>\n- 32-bit and 64-bit support<\/li>\n
- See also: https:\/\/devblogs.microsoft.com\/dotnet\/hardware-intrinsics-in-net-core\/<\/li>\n<\/ul>\n<\/li>\n
- 2020 – .NET 5 – Arm support added to the
System.Runtime.Intrinsics<\/code> namespace\n\n- Introduces
Vector64<T><\/code><\/li>\n- Introduces
AdvSimd<\/code>, ArmBase<\/code>, Dp<\/code>, Rdm<\/code>, Aes<\/code>, Crc32<\/code>, Sha1<\/code>, and Sha256<\/code> for Arm<\/code>\/Arm64<\/code><\/li>\n- Introduces
X86Base<\/code> for x86<\/code>\/x64<\/code><\/li>\n- See also: https:\/\/devblogs.microsoft.com\/dotnet\/announcing-net-5-0-preview-7\/<\/li>\n<\/ul>\n<\/li>\n
- 2021 – .NET 6 – Codegen and infrastructure improvements\n
\n- Introduces
AvxVnni<\/code> for x86<\/code>\/x64<\/code><\/li>\n- Rewrites the
System.Numerics<\/code> implementation to use System.Runtime.Intrinsics<\/code><\/li>\n- See also: https:\/\/devblogs.microsoft.com\/dotnet\/performance-improvements-in-net-6\/<\/li>\n<\/ul>\n<\/li>\n
- 2022 – .NET 7 – Support for writing cross platform algorithms\n
\n- Introduces significant new functionality on the
Vector64<T><\/code>, Vector128<T><\/code>, and Vector256<T><\/code> types that works across platforms<\/li>\n- Introduces
X86Serialize<\/code> for x86<\/code>\/x64<\/code><\/li>\n- Brings the API surface exposed by the above vector types and
Vector<T><\/code> to a parity<\/li>\n- See also: https:\/\/devblogs.microsoft.com\/dotnet\/performance_improvements_in_net_7\/<\/li>\n<\/ul>\n<\/li>\n
- 2023 – .NET 8 – Wasm<\/a> support and AVX-512\n
\n- Introduces
PackedSimd<\/code> and WasmBase<\/code> for Wasm<\/code><\/li>\n- Introduces
Vector512<T><\/code><\/li>\n- Introduces
Avx512F<\/code>, Avx512BW<\/code>, Avx512CD<\/code>, Avx512DQ<\/code>, and Avx512Vbmi<\/code> for x86<\/code>\/x64<\/code><\/li>\n- See also: The rest of this blog post<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Because of this work, with every release .NET libraries and applications gain more power to take advantage of the underlying hardware. In this post I’ll cover in depth what we introduced in .NET 8 and the type of functionality it enables.<\/p>\n
WebAssembly Support<\/h2>\n
WebAssembly<\/a>, or Wasm for short, is essentially code that runs in your browser and which allows a much higher performance profile than typical interpreted scripting support. As a platform, Wasm has started providing underlying SIMD (Single Instruction, Multiple Data) support so that core algorithms can be accelerated and .NET has correspondingly opted to expose support for this functionality via hardware intrinsics.<\/p>\nThis support is very similar to the foundations that other platforms provide and so we won’t go into it in significant detail. Rather, you can simply expect that your existing cross platform algorithms using Vector128<T><\/code> will implicitly light up where supported. If you want to take more direct advantage of functionality that is unique to Wasm, then you can explicitly use the APIs exposed by the PackedSimd<\/code> and WasmBase<\/code> classes in the System.Runtime.Intrinsics.Wasm<\/code> namespace.<\/p>\nAVX-512 Support<\/h2>\n
AVX-512 is a new feature set provided for x86 and x64 computers. It brings along with it a large set of new instructions and hardware functionality that wasn’t previously available including support for 16 additional SIMD registers, dedicated masking, and operating on 512-bits of data at a time. Access to this functionality requires a relatively new processor, namely it requires Skylake-X or newer from Intel and Zen4 or newer from AMD. Because of this, the number of users that can take advantage of this new functionality is smaller, but the improvements it can bring to that hardware are still significant and make it worthwhile to support for data heavy workloads. Additionally, the JIT will opportunistically utilize these instructions for existing SIMD code where it determines benefit to exist. Some examples include:<\/p>\n
\n- using
vpternlog<\/code> instead of and, andn, or<\/code> when a bitwise conditional select is done (Vector128.ConditionalSelect<\/code>)<\/li>\n- using the EVEX encoding to fit more operations into less bytes of code, such as for embedded broadcasts (
x + Vector128.Create(5)<\/code>)<\/li>\n- using newer instructions where support now exists with AVX-512, such as for full-width shuffling and many
long<\/code>\/ulong<\/code> (Int64<\/code>\/UInt64<\/code>) operations<\/li>\n- there were other improvements, that are not listed here, as well and you can expect even more to be added over time\n
\n- some cases such as
Vector<T><\/code> allowing scaling to 512-bits were not completed in .NET 8<\/li>\n<\/ul>\n<\/li>\n<\/ul>\nIn order to support the new vector size of 512-bits, .NET introduced the Vector512<T><\/code> type. This exposes the same general API surface as the other fixed-sized vector types such as Vector256<T><\/code>. It likewise continues exposing the Vector512.IsHardwareAccelerated<\/code> property that allows you to determine whether the general logic should be accelerated in the hardware or if it will end up emulating the behavior via a software fallback.<\/p>\nVector512 is accelerated with AVX-512 by default on Ice Lake and newer hardware (and thus Vector512.IsHardwareAccelerated<\/code> reports true<\/code>), where AVX-512 instructions do not cause the CPU to significantly downclock; where-as utilizing AVX-512 instructions can cause more significant downclocking on Skylake-X, Cascade Lake, and Cooper Lake based hardware (see also 2.5.3 Skylake Server Power Management<\/code> in the Intel\u00ae 64 and IA-32 Architectures Optimization Reference Manual: Volume 1<\/code>). While this is ultimately beneficial for large workloads, it can negatively impact other smaller workloads and as such we default to reporting false<\/code> for Vector512.IsHardwareAccelerated<\/code> on these platforms. Avx512F.IsSupported<\/code> will still report true and the underlying implementation of Vector512<\/code> will still utilize AVX-512<\/code> instructions if called directly. This allows workloads to take advantage of the functionality where they know it to be explicitly beneficial without accidentally causing a negative impact to others.<\/p>\nSpecial Thanks<\/h3>\n
This functionality was made possible via significant contributions by our friends at Intel. The .NET team and Intel have collaborated many times over the years and this continued by us working together on the overall design and implementation, allowing the AVX-512 support to land in .NET 8.<\/p>\n
There was also a great deal of input and validation from the .NET community that helped achieve success and make the release all the better.<\/p>\n
If you would like to contribute or provide input, please join us in the dotnet\/runtime<\/a> repos on GitHub, and tune into API Review on the .NET Foundation YouTube<\/a> channel by following our schedule<\/a> where you can see us discuss new additions to the .NET Libraries and even provide your own input via the chat channel.<\/p>\nIt’s not just 512-bits?<\/h3>\n
Contrary to the name, AVX-512 is not just about 512-bit support. The additional registers, masking support, embedded rounding or broadcast support, and new instructions also all exist for 128-bit and 256-bit vectors. This means that your existing workloads can implicitly get better and you can take explicit advantage of newer functionality where such implicit light up is not possible.<\/p>\n
When SSE was first introduced in 1999 on the Intel Pentium III, it provided 8 registers each 128-bits in length. These registers were known as xmm0<\/code> through xmm7<\/code>. When the x64 platform was later introduced in 2003 on the AMD Athlon 64, it then provided 8 additional registers that were accessible to 64-bit code. These registers were named xmm8<\/code> through xmm15<\/code>. This initial support used a simple encoding scheme that worked in a very similar manner to the general purpose instructions and only allowed for 2 registers to be specified. For something like addition which requires 2 inputs, this meant that one of the registers acted as both an input and an output. This meant that if your input and output needed to be different, you needed 2 instructions to complete the operation. Effectively, your z = x + y<\/code> would become z = x; z += y<\/code>. At the high level these behave the same, but at the low level there is 2 rather than 1 step to make it happen.<\/p>\nThis was then further expanded in 2011 when Intel introduced AVX on the Sandy Bridge based processors by expanding the support to 256-bits. These newer registers were named ymm0<\/code> through ymm15<\/code>, with only registers up to ymm7<\/code> being accessible to 32-bit code. This also introduced a new encoding known as VEX<\/code> (Vector Extensions) which allowed for 3 registers to be encoded. This meant that you could encode z = x + y<\/code> directly and didn’t have to break it into 2 separate steps.<\/p>\nAVX-512 was then introduced by Intel in 2017 with the Skylake-X based processors. This expanded that support to 512-bits and named the registers zmm0<\/code> through zmm15<\/code>. It also introduced 16 new registers, aptly named zmm16<\/code> through zmm31<\/code> and which also have xmm16-xmm31<\/code> and ymm16-ymm31<\/code> variants. As with the previous cases, only registers up to zmm7<\/code> are accessible to 32-bit code. It introduced 8 new registers, named k0<\/code> through k7<\/code>, designed to support “masking” and another new encoding named EVEX<\/code> (Enhanced Vector Extensions) which allows all this new information to be expressed. The EVEX encoding also has other features that allow more common information and operations to be expressed in a more compact fashion. This can help decrease code size while improving performance.<\/p>\nWhat new instructions exist?<\/h3>\n
There is a lot of new functionality, far too much to cover everything in this blog post. But some of the most notable new instructions provide things like:<\/p>\n
\n- Support for doing operations like
Abs<\/code>, Max<\/code>, Min<\/code>, and shifting on 64-bit integers – previously this functionality had to be emulated using multiple instructions<\/li>\n- Support for doing conversions between unsigned integers and floating-point types<\/li>\n
- Support for working with floating-point edge cases<\/li>\n
- Support for fully rearranging the elements in a vector or multiple vectors<\/li>\n
- Support for doing 2 bitwise operations in a single instruction<\/li>\n<\/ul>\n
The 64-bit integer support is notable because it means working with 64-bit data doesn’t need to use a slower or alternative code sequence to support the same functionality. It makes it much easier to just write your code and expect it to behave the same regardless of the underlying data type you’re working with.<\/p>\n
The floating-point to unsigned integer conversion support is notable for similar reasons. Converting from double<\/code> to long<\/code> required a single instruction, but converting from double<\/code> to ulong<\/code> required many instructions. With AVX-512 this becomes a single instruction and allows users to get the expected performance when working with unsigned data. This can be common in various image processing or Machine Learning scenarios.<\/p>\nThe expanded support for floating-point data is one of my favorite features of AVX-512. Some examples include the ability to extract the unbiased exponent (Avx512F.GetExponent<\/code>) or the normalized mantissa (Avx512F.GetMantissa<\/code>), to round a floating-point value to a specific number of fraction bits (Avx512F.RoundScale<\/code>), to multiply a value by 2^x (Avx512F.Scale<\/code>, known in C as scalebn<\/code>), to perform Min<\/code>, Max<\/code>, MinMagnitude<\/code>, and MaxMagnitude<\/code> with correct handling of +0<\/code> and -0<\/code> (Avx512DQ.Range<\/code>), and even to perform reductions which are useful when handling large values for trigonometric functions like Sin<\/code> or Cos<\/code> (Avx512DQ.Reduce<\/code>).<\/p>\nHowever, one of my personal favorites is an instruction named vfixupimm<\/code> (Avx512F.Fixup<\/code>). At a high level, this instruction lets you detect a number of input edge cases and “fixup” the output to be one of the common outputs and to do this per element. This can massively improve the performance of some algorithms and greatly reduces the amount of handling required. The way it works is it takes 4 inputs known as left<\/code>, right<\/code>, table<\/code>, and control<\/code>. It first does a classification of the floating-point value in right<\/code> and determines if it is QNaN<\/code> (0), SNaN<\/code> (1), +\/-0<\/code> (2), +1<\/code> (3), -Infinity<\/code> (4), +Infinity<\/code> (5), Negative<\/code> (6), or Positive<\/code> (7). It then uses that to read 4<\/code> bits from table<\/code> (QNaN<\/code> being 0<\/code>, reads bits 0..3<\/code>; Negative<\/code> being 6<\/code> reads bits 24..27<\/code>). The value of those 4 bits in table<\/code> then determines what the result will be. The possible results (per element) are:<\/p>\n\n\n\nBit Pattern<\/th>\n Definition<\/th>\n<\/tr>\n<\/thead>\n \n\n0b0000<\/td>\n left[i]<\/td>\n<\/tr>\n \n0b0001<\/td>\n right[i]<\/td>\n<\/tr>\n \n0b0010<\/td>\n QNaN(right[i])<\/td>\n<\/tr>\n \n0b0011<\/td>\n QNaN<\/td>\n<\/tr>\n \n0b0100<\/td>\n -Infinity<\/td>\n<\/tr>\n \n0b0101<\/td>\n +Infinity<\/td>\n<\/tr>\n \n0b0110<\/td>\n IsNegative(right[i]) ? -Infinity : +Infinity<\/td>\n<\/tr>\n \n0b0111<\/td>\n -0.0<\/td>\n<\/tr>\n \n0b1000<\/td>\n +0.0<\/td>\n<\/tr>\n \n0b1001<\/td>\n -1.0<\/td>\n<\/tr>\n \n0b1010<\/td>\n +1.0<\/td>\n<\/tr>\n \n0b1011<\/td>\n +0.5<\/td>\n<\/tr>\n \n0b1100<\/td>\n +90.0<\/td>\n<\/tr>\n \n0b1101<\/td>\n PI \/ 2<\/td>\n<\/tr>\n \n0b1110<\/td>\n MaxValue<\/td>\n<\/tr>\n \n0b1111<\/td>\n MinValue<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nWith SSE there was some support for rearranging the data in a vector. Say, for example, you had 0, 1, 2, 3<\/code> and you wanted it ordered 3, 1, 2, 0<\/code>. With the introduction of AVX and the expansion to 256-bits, this support was likewise expanded. However, due to how the instructions operated you’d actually do the same 128-bit operation twice. This made it simple to expand existing algorithms to 256-bits since you effectively are just doing the same thing twice. However, it made working with other algorithms more difficult when you actually needed to consider the entire vector cohesively. There were some instructions that let you rearrange the data across the entire 256-bit vector, but they were often limited either in how the data could be rearranged or in the types they supported (full shuffle of byte elements is a notable example of missing support). AVX-512 has many of the same considerations for its expanded 512-bit support. However, it also introduces new instructions that fill the gap and now let you fully rearrange the elements for any size of element.<\/p>\nFinally, one of my other personal favorites is an instruction named vpternlog<\/code> (Avx512F.TernaryLogic<\/code>). This instruction lets you take any 2 bitwise operations and combine them, so they can be executed in a single instruction. For example, you can do (a & b) | c<\/code>. The way it works is that it takes 4 inputs, a<\/code>, b<\/code>, c<\/code>, and control<\/code>. You then have 3 keys to remember: A: 0xF0<\/code>, B: 0xCC<\/code>, C: 0xAA<\/code>. In order to represent the operation desired, you simply build the control<\/code> by performing that operation on those keys. So, if you wanted to simply return a<\/code>, you’d use 0xF0<\/code>. If you wanted to do a & b<\/code>, you’d use (byte)(0xF0 & 0xCC)<\/code>. If you wanted to do (a & b) | c<\/code>, then it is (byte)((0xF0 & 0xCC) | 0xAA<\/code>. There are 256 different operations possible in total, with the basic building blocks being those keys and the following bitwise operations:<\/p>\n\n\n\nOperation<\/th>\n Definition<\/th>\n<\/tr>\n<\/thead>\n \n\nnot<\/td>\n ~x<\/td>\n<\/tr>\n \nand<\/td>\n x & y<\/td>\n<\/tr>\n \nnand<\/td>\n ~x & y<\/td>\n<\/tr>\n \nor<\/td>\n x<\/td>\n y<\/td>\n<\/tr>\n \nnor<\/td>\n ~x<\/td>\n y<\/td>\n<\/tr>\n \nxor<\/td>\n x ^ y<\/td>\n<\/tr>\n \nxnor<\/td>\n ~x ^ y<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nThere are then some special operations that are also supported given the above basic operations and which can expand even further.<\/p>\n
\n\n\nOperation<\/th>\n Definition<\/th>\n<\/tr>\n<\/thead>\n \n\nfalse<\/td>\n Bit pattern of 0x00<\/td>\n<\/tr>\n \ntrue<\/td>\n Bit pattern of 0xFF<\/td>\n<\/tr>\n \nmajor<\/td>\n Returns 0 if two or more input bits are 0, returns 1 if two or more input bits are 1<\/td>\n<\/tr>\n \nminor<\/td>\n Returns 0 if two or more input bits are 1, returns 1 if two or more input bits are 0<\/td>\n<\/tr>\n \nconditional select<\/td>\n Logically (x & y) | (~x & z)<\/code>, which works since it is (x and y) or (x nand y)<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\nIn .NET 8 we didn’t complete the support to implicitly recognize and fold these patterns to emit vpternlog<\/code>. We expect that to debut in .NET 9.<\/p>\nWhat is masking support?<\/h3>\n
At the simplest level, writing vectorized code involves using SIMD to do the same basic operation on Count<\/code> different elements of type T<\/code> in a single instruction. This works very nicely when the same operation needs to be done to all data. However, not all data is necessarily uniform and sometimes you need to handle particular inputs differently. For example, you may want to do a different operation for positive vs negative numbers. You may need to return a different result if the user has passed in NaN<\/code>, and so on. When writing regular code, you would normally handle this with a branch and this works very nicely. When writing vectorized code, however, such branches break the ability to use SIMD instructions since you have to handle each element independently. .NET takes advantage of this in various locations, including the new TensorPrimitives<\/code> APIs where it allows us to handle trailing data that would otherwise not fit into a full vector.<\/p>\nThe typical solution for this is to write “branch-free” code. One of the simplest ways to do this is to compute both answers and then use bitwise operations to pick the correct answer. You can think of this a lot like a ternary condition