The .NET team has significantly improved performance with .NET 5, both generally and for ARM64. You can check out the general improvements in the excellent and detailed Performance Improvements in .NET 5 blog by Stephen. In this post, I will describe the performance improvements we made specifically for ARM64 and show the positive impact on the benchmarks we use. I will also share some of the additional opportunities for performance improvements that we have identified and plan to address in a future release.

While we have been working on ARM64 support in RyuJIT for over five years, most of the work that was done was to ensure that we generate functionally correct ARM64 code. We spent very little time in evaluating the performance of the code RyuJIT produced for ARM64. As part of .NET 5, our focus was to perform investigation in this area and find out any obvious issues in RyuJIT that would improve the ARM64 code quality (CQ). Since Microsoft VC++ team already has support for Windows ARM64, we consulted with them to understand the CQ issues that they encountered when doing a similar exercise.

Although fixing CQ issues is crucial, sometimes its impact might not be noticeable in an application. Hence, we also wanted to make observable improvements in the performance of .NET libraries to benefit .NET applications targeted for ARM64.

Here is the outline I will use to describe our work for improving ARM64 performance on .NET 5:

• ARM64-specific optimizations in the .NET libraries.
• Evaluation of code quality produced by RyuJIT and resulting outcome.

ARM64 hardware intrinsics in .NET libraries

In .NET Core 3.0, we introduced a new feature called “hardware intrinsics” which gives access to various vectorized and non-vectorized instructions that modern hardware support. .NET developers can access these instructions using set of APIs under namespace System.Runtime.Intrinsics and System.Runtime.Intrinsics.X86 for x86/x64 architecture. In .NET 5, we added around 384 APIs under System.Runtime.Intrinsics.Arm for ARM32/ARM64 architecture. This involved implementing those APIs and making RyuJIT aware of them so it can emit appropriate ARM32/ARM64 instruction. We also optimized methods of Vector64 and Vector128 that provide ways to create and manipulate Vector64<T> and Vector128<T> datatypes on which majority of the hardware intrinsic APIs operate on. If interested, refer to the sample code usage along with examples of Vector64 and Vector128 methods here. You can check our “hardware intrinsic” project progress here.

Optimized .NET library code using ARM64 hardware intrinsics

In .NET Core 3.1, we optimized many critical methods of .NET library using x86/x64 intrinsics. Doing that improved the performance of such methods when ran on hardware supporting the x86/x64 intrinsic instructions. For hardware that does not support x86/x64 intrinsics such as ARM machines, .NET would fallback to the slower implementation of those methods. dotnet/runtime#33308 list such .NET library methods. In .NET 5, we have optimized most of these methods using ARM64 hardware intrinsics as well. So, if your code uses any of those .NET library methods, they will now see speed boost running on ARM architecture. We focused our efforts on methods that already were optimized with x86/x64 intrinsics, because those were chosen based on an earlier performance analysis (which we didn’t want to duplicate/repeat) and we wanted the product to have generally similar behavior across platforms. Moving forward, we expect to use both x86/x64 and ARM64 hardware intrinsics as our default approach when we optimize .NET library methods. We still have to decide how this will affect our policy for PRs that we accept.

For each of the methods that we optimized in .NET 5, I’ll show you the improvements in terms of the low-level benchmark that we used for validating our improvements. These benchmarks are far from real-world. You’ll see later in the post how all of these targeted improvements combine together to greatly improve .NET on ARM64 in larger, more real-world, scenarios.

System.Collections

System.Collections.BitArray methods were optimized by @Gnbrkm41 in dotnet/runtime#33749. The following measurements are in nanoseconds for Perf_BitArray microbenchmark.

System.Numerics

System.Numerics.BitOperations methods were optimized in dotnet/runtime#34486 and dotnet/runtime#35636. The following measurements are in nanoseconds for Perf_BitOperations microbenchmark.

System.Numerics.Matrix4x4 methods were optimized in dotnet/runtime#40054. The following measurements are in nanoseconds for Perf_Matrix4x4 microbenchmark.

The SIMD accelerated types System.Numerics.Vector2, System.Numerics.Vector3 and System.Numerics.Vector4 were optimized in dotnet/runtime#35421, dotnet/runtime#36267, dotnet/runtime#36512, dotnet/runtime#36579 and dotnet/runtime#37882 to use hardware intrinsics. The following measurements are in nanoseconds for Perf_Vector2, Perf_Vector3 and Perf_Vector4 microbenchmarks.

System.SpanHelpers

System.SpanHelpers methods were optimized in dotnet/runtime#37624 and dotnet/runtime#37934 work. The following measurements are in nanoseconds for Span<T>.IndexOfValue and ReadOnlySpan.IndexOfString microbenchmarks.

System.Text

We have also optimized methods in several classes under System.Text.

• Methods in System.Text.ASCIIUtility were optimized in dotnet/runtime#38597 and dotnet/runtime#39506.
• System.Text.Unicode were optimized in dotnet/runtime#38653, dotnet/runtime#39041 and dotnet/runtime#39050
• System.Text.Encodings.Web were optimized in dotnet/runtime#38707

In .NET 6, we are planning to optimize remaining methods of System.Text.ASCIIUtility described in dotnet/runtime#41292, methods of System.Buffers to address dotnet/runtime#35033 and merge the work to optimize JsonReaderHelper.IndexOfLessThan done by Ben Adams in dotnet/runtime#41097.

All the measurements that I have mentioned above came from our performance lab runs done on Ubuntu machines on 8/6/2020, 8/10/2020 and 8/28/2020.

Details

It is probably clear at this point how impactful and important hardware intrinsics are. I want to show you more, by walking through an example. Imagine a Test() returns leading zero count of argument value.

private int Test(uint value)
{
    return BitOperations.LeadingZeroCount(value);
}

Before optimization for ARM64, the code would execute the software fallback of LeadingZeroCount(). If you see the ARM64 assembly code generated below, not only it is large, but RyuJIT had to JIT 2 methods – Test(int) and Log2SoftwareFallback(int).

; Test(int):int

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        cbnz    w0, M00_L00
        mov     w0, #32
        b       M00_L01
M00_L00:
        bl      System.Numerics.BitOperations:Log2SoftwareFallback(int):int
        eor     w0, w0, #31
M00_L01:
        ldp     fp, lr, [sp],#16
        ret     lr

; Total bytes of code 28, prolog size 8
; ============================================================


; System.Numerics.BitOperations:Log2SoftwareFallback(int):int

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        lsr     w1, w0, #1
        orr     w0, w0, w1
        lsr     w1, w0, #2
        orr     w0, w0, w1
        lsr     w1, w0, #4
        orr     w0, w0, w1
        lsr     w1, w0, #8
        orr     w0, w0, w1
        lsr     w1, w0, #16
        orr     w0, w0, w1
        movz    w1, #0xacdd
        movk    w1, #0x7c4 LSL #16
        mul     w0, w0, w1
        lsr     w0, w0, #27
        sxtw    x0, w0
        movz    x1, #0xc249
        movk    x1, #0x5405 LSL #16
        movk    x1, #0x7ffc LSL #32
        ldrb    w0, [x0, x1]
        ldp     fp, lr, [sp],#16
        ret     lr

; Total bytes of code 92, prolog size 8

After we optimized LeadingZeroCount() to use ARM64 intrinsics, generated code for ARM64 is just a handful of instructions (including the crucial clz). In this case, RyuJIT did not even JIT Log2SoftwareFallback(int) method because it was not called. Thus, by doing this work, we got improvement in code quality as well as JIT throughput.

; Test(int):int

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        clz     w0, w0
        ldp     fp, lr, [sp],#16
        ret     lr

; Total bytes of code 24, prolog size 8

AOT compilation for methods having ARM64 intrinsics

In the typical case, applications are compiled to machine code at runtime using the JIT. The target machine code produced is very efficient but has the disadvantage of having to do the compilation during execution and this might add some delay during the application start-up. If the target platform is known in advance, you can create ready-to-run (R2R) native images for that target platform. This is known as ahead of time (AOT) compilation. It has the advantage of faster startup time because there is no need to produce machine code during execution. The target machine code is already present in the binary and can be run directly. AOT compiled code might be suboptimal sometimes but get replaced by optimal code eventually.

Until .NET 5, if a method (.NET library method or user defined method) had calls to ARM64 hardware intrinsic APIs (APIs under System.Runtime.Intrinsics and System.Runtime.Intrinsics.Arm), such methods were never compiled AOT and were always deferred to get compiled during runtime. This had an impact on start-up time of some .NET apps which used one of these methods in their startup code. In .NET 5, we addressed this problem in dotnet/runtime#38060 and now able to do the compilation of such methods AOT.

Microbenchmark analysis

Optimizing the .NET libraries with intrinsics was a straightforward step (following in the path of what we’d already done for x86/x64). An equal or more significant project was improving the quality of code that the JIT generates for ARM64. It’s important to make that exercise data-oriented. We picked benchmarks that we thought would highlight underlying ARM64 CQ issues. We started with the Microbenchmarks that we maintain. There are around 1300 of these benchmarks.

We compared ARM64 and x64 performance numbers for each of these benchmarks. Parity was not our goal, however, it is always useful to have a baseline to compare with, particularly to identify outliers. We then identified the benchmarks with the worst performance, and determined why that was the case. We tried using some profilers like WPA and PerfView but they were not useful in this scenario. Those profilers would have pointed out the hottest method in given benchmark. But since MicroBenchmarks are tiny benchmarks with at most 1~2 methods, the hottest method that the profiler pointed was mostly the benchmark method itself. Hence, to understand the ARM64 CQ issues, we decided to just inspect the assembly code produced for a given benchmark and compare it with x64 assembly. That would help us identify basic issues in RyuJIT’s ARM64 code generator.

Next, I will describe some of the issues that we found with this exercise.

Memory barriers in ARM64

Through some of the benchmarks, we noticed accesses of volatile variables in hot loop of critical methods of System.Collections.Concurrent.ConcurrentDictionary class. Accessing volatile variable for ARM64 is expensive because they introduce memory barrier instructions. I’ll describe why, shortly. By caching the volatile variable and storing it in a local variable (dotnet/runtime#34225, dotnet/runtime#36976 and dotnet/runtime#37081) outside the loop resulted in improved performance, as seen below. All the measurements are in nanoseconds.

We made similar optimization in System.Threading.ThreadPool as part of dotnet/runtime#36697 and in System.Diagnostics.Tracing.EventCount as part of dotnet/runtime#37309 classes.

ARM memory model

ARM architecture has weakly ordered memory model. The processor can re-order the memory access instructions to improve performance. It can rearrange instructions to reduce the time processor takes to access memory. The order in which instructions are written is not guaranteed, and instead may be executed depending on the memory access cost of a given instruction. This approach does not impact single core machine but can negatively impact a multi-threaded program running on a multicore machine. In such situations, there are instructions to tell processors not to re-arrange memory access at a given point. The technical term for such instructions that restricts this re-arrangement is called “memory barriers”. The dmb instruction in ARM64 acts as a barrier prohibiting the processor from moving an instruction across the fence. You can read more about it in ARM developer docs.

One of the way in which you can specify adding memory barrier in your code is by using a volatile variable. With volatile, it is guaranteed that the runtime, JIT, and the processor will not rearrange reads and writes to memory locations for performance. To make this happen, RyuJIT will emit dmb (data memory barrier) instruction for ARM64 every time there is an access (read/write) to a volatile variable.

For example, the following is code taken from Perf_Volatile microbenchmark. It does a volatile read of local field _location.

public class Perf_Volatile
{
    private double _location = 0;
    
    [Benchmark]
    public double Read_double() => Volatile.Read(ref _location);
}

The generated relevant machine code of Read_double for ARM64 is:

; Read_double():double:this

        add     x0, x0, #8
        ldr     d0, [x0]
        dmb     ishld

The code first gets the address of _location field, loads the value in d0 register and then execute dmb ishld that acts as a data memory barrier.

Although this guarantees the memory ordering, there is a cost associated with it. The processor must now guarantee that all the data access done before the memory barrier is visible to all the cores after the barrier instruction which could be time consuming. Hence, it is important to avoid or minimize the usage of such data access inside hot methods and loop as much as possible.

ARM64 and big constants

In .NET 5, we did some improvements in the way we handled large constants present in user code. We started eliminating redundant loads of large constants in dotnet/runtime#39096 which gave us around 1% (521K bytes to be precise) improvement in the size of ARM64 code we produced for all the .NET libraries.

It is worth noting that sometimes JIT improvements do not get reflected in the microbenchmark runs but are beneficial in overall code quality. In such cases, RyuJIT team reports the improvements that were made in terms of .NET libraries code size. RyuJIT is run on entire .NET library dlls before and after changes to understand how much impact the optimization has made, and which libraries got optimized more than others. As of preview 8, the emitted code size of entire .NET libraries for ARM64 target is 45 MB. 1% improvement would mean we emit 450 KB less code in .NET 5, which is substantial. You can see the individual numbers of methods that were improved here.

Details

ARM64 has an Instruction set architecture (ISA) with fixed length encoding with each instruction exactly 32-bits long. Because of this, a move instruction mov have space only to encode up to 16-bits unsigned constant. To move a bigger constant value, we need to move the value in multiple steps using chunks of 16-bits (movz/movk). Due to this, multiple mov instructions are generated to construct a single bigger constant that need to be saved in a register. Alternately, in x64 a single mov can load bigger constant.

Now imagine code containing a couple constants (2981231 and 2981235).

public static uint GetHashCode(uint a, uint b)
{
  return  ((a * 2981231) * b) + 2981235;
}

Before we optimized this pattern, we would generate code to construct each constant. So, if they are present in a loop, they would get constructed for every iteration.

        movz    w2, #0x7d6f
        movk    w2, #45 LSL #16  ; <-- loads 2981231 in w2
        mul     w0, w0, w2
        mul     w0, w0, w1
        movz    w1, #0x7d73
        movk    w1, #45 LSL #16  ; <-- loads 2981235 in w1
        add     w0, w0, w1

In .NET 5, we are now loading such constants once in a register and whenever possible, reusing them in the code. If there is more than one constant whose difference with the optimized constant is below a certain threshold, then we use the optimized constant that is already in a register to construct the other constant(s). Below, we used the value in register w2 (2981231 in this case) to calculate constant 2981235.

        movz    w2, #0x7d6f
        movk    w2, #45 LSL #16  ; <-- loads 2981231
        mul     w0, w0, w2
        mul     w0, w0, w1
        add     w1, w2, #4       ; <-- loads 2981235
        add     w0, w0, w1

This optimization was helpful not just for loading constants but also for loading method addresses because they are 64-bits long on ARM64.

C# structs

We made good progress in optimizing scenarios for ARM64 that returns C# struct and got 0.19% code size improvement in .NET libraries. Before .NET 5, we always created a struct on stack before doing any operation on it. Any updates to its fields would do the update on stack. When returning, the fields had to be copied from the stack into the return register. Likewise, when a struct was returned from a method, we would store it on stack before operating on it. In .NET 5, we started enregistering structs that can be returned using multiple registers in dotnet/runtime#36862, meaning in certain cases, the structs won’t be created on stack but will be directly created and manipulated using registers. With that, we omitted the expensive memory access in methods using structs. This was substantial work that improved scenarios that operate on stack.

The following measurements are in nanoseconds for ReadOnlySpan<T> and Span<T> .ctor() microbenchmark that operates on ReadOnlySpan<T> and Span<T> structs.

Details

In .NET Core 3.1, when a function created and returned a struct containing fields that can fit in a register like float, we were always creating and storing the struct on stack. Let us see an example:

public struct MyStruct
{
  public float a;
  public float b;
}

[MethodImpl(MethodImplOptions.NoInlining)]
public static MyStruct GetMyStruct(float i, float j)
{
  MyStruct mys = new MyStruct();
  mys.a = i + j;
  mys.b = i - j;
  return mys;
}

public static float GetTotal(float i, float j)
{
  MyStruct mys = GetMyStruct(i, j);
  return mys.a + mys.b;
}

public static void Main()
{
  GetTotal(1.5f, 2.5f);
}

Here is the code we generated in .NET Core 3.1. If you see below, we created the struct on stack at location [fp+24] and then stored the i+j and i-j result in fields a and b located at [fp+24] and [fp+28] respectively. We finally loaded those fields from stack into the registers s0 and s1 to return the result. The caller GetTotal() would also save the returned struct on stack before operating on it.

; GetMyStruct(float,float):struct

        stp     fp, lr, [sp,#-32]!
        mov     fp, sp
        str     xzr, [fp,#24]	
        add     x0, fp, #24   ; <-- struct created on stack at [fp+24]
        str     xzr, [x0]
        fadd    s16, s0, s1
        str     s16, [fp,#24] ; <-- mys.a = i + j
        fsub    s16, s0, s1
        str     s16, [fp,#28] ; <-- mys.a = i - j
        ldr     s0, [fp,#24]  ; returning the struct field 'a' in s0
        ldr     s1, [fp,#28]  ; returning the struct field 'b' in s1
        ldp     fp, lr, [sp],#32
        ret     lr

; Total bytes of code 52, prolog size 12
; ============================================================

; GetTotal(float,float):float

        stp     fp, lr, [sp,#-32]!
        mov     fp, sp
        call    [GetMyStruct(float,float):MyStruct]
        str     s0, [fp,#24]   ; store mys.a on stack
        str     s1, [fp,#28]   ; store mys.b on stack
        add     x0, fp, #24    
        ldr     s0, [x0]       ; load again in register
        ldr     s16, [x0,#4]
        fadd    s0, s0, s16
        ldp     fp, lr, [sp],#32
        ret     lr

; Total bytes of code 44, prolog size 8

With the enregistration work, we do not create the struct on stack anymore in certain scenarios. With that, we do not have to load the field values from stack into the return registers. Here is the optimized code in .NET 5:

; GetMyStruct(float,float):MyStruct

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        fadd    s16, s0, s1
        fsub    s1, s0, s1   ; s1 contains value of 'b'
        fmov    s0, s16      ; s0 contains value of 'a'
        ldp     fp, lr, [sp],#16
        ret     lr


; Total bytes of code 28, prolog size 8
; ============================================================

; GetTotal(float,float):float

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        call    [GetMyStruct(float,float):MyStruct]
        fmov    s16, s1
        fadd    s0, s0, s16
        ldp     fp, lr, [sp],#16
        ret     lr

; Total bytes of code 28, prolog size 8

The code size has reduced by 43% and we have eliminated 10 memory accesses in GetMyStruct() and GetTotal() combined. The stack space needed for both the methods has also reduced from 32 bytes to 16 bytes.

dotnet/runtime#39326 is a work in progress to similarly optimize fields of structs that are passed in registers, that we will ship in next release. We also found issues like dotnet/runtime#35071 where we do some redundant store and load when handling struct arguments or HFA registers, or always push arguments on the stack before using them in a method as seen in dotnet/runtime#35635. We are hoping to address these issues in a future release.

Array access with post-index addressing mode

ARM64 has various addressing modes that can be used to generate load/store instruction to compute the memory address an operation need to access. “Post-index” addressing mode is one of them. It is usually used in scenarios where consecutive access to memory location (from fixed base address) is needed. A typical example of it is array element access in a loop where the base address of an array is fixed and the elements are in consecutive memory at a fixed offset from one another. One of the issues we found out was that we were not using post-index addressing mode in our generated ARM64 code but instead generating a lot of instructions to calculate the address of array element. We will address dotnet/runtime#34810 in a future release.

Details

Consider a loop that stores a value in an array element.

public int[] Test()
{
    int[] arr = new int[10];
    int i = 0;
    while (i < 9)
    {
        arr[i] = 1;  // <---- IG03
        i++;
    }
    return arr;
}

To store 1 inside arr[i], we need to generate instructions to calculate address of arr[i] in every iteration. For example, on x64 this is as simple as:

...
M00_L00:
        movsxd   rcx, edx
        mov      dword ptr [rax+4*rcx+16], 1
        inc      edx
        cmp      edx, 9
        jl       SHORT M00_L00
...

rax stores the base address of array arr. rcx holds the value of i and since the array is of type int, we multiply it by 4. rax+4*rcx forms the address of array element at ith index. 16 is the offset from base address at which elements are stored. All of this execute in a loop.

However, for ARM64, we generate longer code as seen below. We generate 3 instructions to calculate the array element address and 4th instruction to save the value. We do this calculation in every iteration of a loop.

...
M00_L00:
        sxtw    x2, w1        ; load 'i' from w1
        lsl     x2, x2, #2    ; x2 *= 4
        add     x2, x2, #16   ; x2 += 16
        mov     w3, #1        ; w3 = 1
        str     w3, [x0, x2]  ; store w3 in [x0 + x2]
        add     w1, w1, #1    ; w1++
        cmp     w1, #9        ; repeat while i < 9
        blt     M00_L00
...

With post-index addressing mode, much of the recalculation here can be simplified. With this addressing mode, we can auto increment the address present in a register to get the next array element. The code gets optimized as seen below. After every execution, contents of x1 would be auto incremented by 4, and would get the address of the next array element.

; x1 contains <<base address of arr>>+16
; w0 contains value "1"
; w1 contains value of "i"

M00_L00:
        str     w0, [x1], 4  ; post-index addressing mode
        add     w1, w1, #1
        cmp     w1, #9
        blt     M00_L00

Fixing this issue will result in both performance and code size improvements.

Mod operations

Modulo operations are crucial in many algorithms and currently we do not generate good quality code for certain scenarios. In a % b, if a is an unsigned int and b is power of 2 and a constant, ARM64 code that is generated today is:

        lsr     w1, w0, #2
        lsl     w1, w1, #2
        sub     w0, w0, w1

But instead it can be optimized to generate:

        and     w2, w0, <<b - 1>>

Another scenario that we could optimize is if b is a variable. Today, we generate:

        udiv    w2, w0, w1   ; sdiv if 'a' is signed int
        mul     w1, w2, w1
        sub     w0, w0, w1

The last two instructions can be combined into a single instruction to generate:

        udiv    w2, w0, w1
        msub    w3, w3, w1, w2

We will address dotnet/runtime#34937 in a future release.

Code size analysis

Understanding the size of ARM64 code that we produced and reducing it down was an important task for us in .NET 5. Not only does it improve the memory consumption of .NET runtime, it also reduces the disk footprint of R2R binaries that are compiled ahead-of-time.

We found some good areas where we could reduce the ARM64 code size and the results were astonishing. In addition to some of the work I mentioned above, after we optimized code generated for call indirects in dotnet/runtime#35675 and virtual call stub in dotnet/runtime#36817, we saw code size improvement of 13% on .NET library R2R images. We also compared the ARM64 code produced in .NET Core 3.1 vs. .NET 5 for the top 25 NuGet packages. On average, we improved the code size of R2R images by 16.61%. Below are the nuget package name and version along with the % improvement. All the measurements are in bytes (lower is better).

Note that most of the above packages might not include R2R images, we picked these packages for our code size measurement because they are one of the most downloaded packages and written for wide variety of domains.

Inline heuristics tweaking

Currently, RyuJIT uses various heuristics to decide whether inlining a method will be beneficial or not. Among other heuristics, one of them is to check the code size of the caller in which the callee gets inlined. The code size heuristics is based upon x64 code which has different characteristics than the ARM64 code. We explored some ways to fine tune it for ARM64 but did not see promising results. We will continue exploring these heuristics in future.

Return address hijacking

While doing the code size analysis, we noticed that for small methods, ARM64 code includes prologue and epilogue for every method, even though it is not needed. Often small methods get inlined inside the caller, but there may be scenarios where this might not happen. Consider a method AdditionalCount() that is marked as NoInlining. This method will not get inlined inside its caller. In this method, let us invoke the Stack<T>.Count getter.

[MethodImpl(MethodImplOptions.NoInlining)]
public static int AdditionalCount(Stack<string> a, int b)
{
    return a.Count + b;
}

Since there are no local variables in AdditionalCount(), nothing is retrieved from the stack and hence there is no need prepare and revert stack’s state using prologue and epilogue. Below is the code generated for x64. If you notice, the x64 code for this method is 6 bytes long, with 0 bytes in prolog.

; AdditionalCount(System.Collections.Generic.Stack`1[[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]],int):int

        mov      eax, edx
        add      eax, dword ptr [rcx+16]
        ret

; Total bytes of code 6, prolog size 0

However, for ARM64, we generate prologue and epilogue even though nothing is stored or retrieved from stack. Also, if you see below, the code size is 24 bytes with 8 bytes in prologue which is bigger than x64 code size.

; AdditionalCount(System.Collections.Generic.Stack`1[[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]],int):int

        stp     fp, lr, [sp,#-16]!
        mov     fp, sp
        ldr     w0, [x0,#16]
        add     w0, w0, w1
        ldp     fp, lr, [sp],#16
        ret     lr

; Total bytes of code 24, prolog size 8

Our investigation showed that approximately 23% of methods in the .NET libraries skip generating prologue/epilogue for x64, while for ARM64, we generate extra 16 bytes code for storing and retrieving fp and lr registers. We need to do this to support return address hijacking. If the .NET runtime needs to trigger garbage collection (GC), it needs to bring the user code execution to a safe point before it can start the GC. For ARM64, it has been done by generating prologue/epilogue in user’s code to store the return address present in lr register on the stack and retrieve it back before returning. If the runtime decides to trigger GC while executing user code, it replaces the return address present on the stack with a runtime helper function address. When the method completes the execution, it retrieves the modified return address from the stack into lr and thus return to the runtime helper function so the runtime can perform GC. After GC is complete, control jumps back to the original return address of user code. All this is not needed for x64 code because the return address is already on stack and can be retrieved by the runtime. It may be possible to optimize return address hijacking for certain scenarios. In future release, we will do more investigation of dotnet/runtime#35274 to reduce the code size and improve speed of small methods.

ARM64 code characteristics

Although there are various issues that we have identified and continue optimizing to improve the code size produced for ARM64, there are certain aspects of ARM ISA that cannot be changed and are worth mentioning here.

While x86 has CISC and ARM is a RISC architecture, it is nearly impossible to have x86 and ARM target code size similar for the same method. ARM has fixed length encoding of 4-bytes in contrast to x86 which has variable length encoding. A return instruction ret on x86 can be as short as 1-byte, but on ARM64, it is always 4-bytes long. Because of fixed length encoding in ARM, there is a limited range of constant values that can be encoded inside an instruction as I mentioned in ARM64 and big constants section. Any instruction that contains a constant bigger than 12-bits (sometimes 16-bits) must be moved to a register and operated through register. Basic arithmetic instructions like add and sub cannot operate on constant values that are bigger than 12-bits. Data cannot be transferred between memory to memory. It must be loaded in a register before transferring or operating on it. If there are any constants that need to be stored in memory, those constants must be moved in a register first before storing them to the memory. Even to do memory access using various addressing modes, the address has to be moved in a register before loading or storing data into it. Thus, at various places, there is a need to perform prerequisite or setup instructions to load the data in registers before performing actual operation. That all can lead to bigger code size on ARM64 targets.

Peephole analysis

The last topic that I would like to mention is our data-driven engineering approach in discovering and prioritizing some other important ARM64 code quality enhancements. When inspecting ARM64 code produced for .NET libraries with several benchmarks, we realized that there were several instruction patterns that could be replaced with better and more performant instructions. In compiler literature, “peephole optimization” is the phase that does such optimizations. RyuJIT does not have peephole optimization phase currently. Adding a new compiler phase is a big task and can easily take a few months to get it right without impacting other metrics like JIT throughput. Additionally, we were not sure how much code size or speed up improvement such optimization would get us. Hence, we gathered data in an interesting way to discover and prioritize various opportunities in performing peephole optimization. We wrote a utility tool AnalyzeAsm that would scan through approximately 1GB file containing ARM64 disassembly code of .NET library methods and report back the frequency of instruction patterns that we were interested in, along with methods in which they are present. With that information, it became easier for us to decide that a minimal implementation of peephole optimization phase was important. With AnalyzeAsm, we identified several peephole opportunities that would give us roughly 0.75% improvement in the code size of the .NET libraries. In .NET 5, we optimized an instruction pattern by eliminating redundant opposite mov instructions in dotnet/runtime#38179 which gave us 0.28% code size improvement. Percentage-wise, the improvements are not large, but they are meaningful in the context of the whole product.

Details

I would like to highlight some of the peephole opportunities that we have found and hoping to address them in .NET 6.

Replace pair of “ldr” with “ldp”

If there are pair of consecutive load instructions ldr that loads data into a register from consecutive memory location, then the pair can be replaced by single load-pair instruction ldp.

So below pattern:

        ldr     x23, [x19,#16]
        ldr     x24, [x19,#24]

can be replaced with:

        ldp     x23, x24, [x19, #16]

As seen in dotnet/runtime#35130 and dotnet/runtime#35132, AnalyzeAsm pointed out that this pattern occurs approximately 34,000 times in 16,000 methods.

Replace pair of “str” with “stp”

This is similar pattern as above, except that if there are pair of consecutive store instructions str that stores data from a register into consecutive memory location, then the pair can be replaced by single store-pair instruction stp.

So below pattern:

        str     x23, [x19,#16]
        str     x24, [x19,#24]

can be replaced with:

        stp     x23, x24, [x19, #16]

As seen in dotnet/runtime#35133 and dotnet/runtime#35134, AnalyzeAsm pointed out that this pattern occurs approximately 35,000 times in 16,400 methods.

Replace pair of “str wzr” with “str xzr”

wzr is 4-byte zero register while xzr is an 8-byte zero register in ARM64. If there is a pair of consecutive instructions that stores wzr in consecutive memory location, then the pair can be replaced by single store of xzr value.

So below pattern:

        str     wzr, [x2, #8]
        str     wzr, [x2, #12]

can be replaced with:

        str     xzr, [x2, #8]

As seen in dotnet/runtime#35136, AnalyzeAsm pointed out that this pattern occurs approximately 450 times in 353 methods.

Remove redundant “ldr” and “str”

Another pattern that we were generating was loading a value from memory location into a register and then storing that value back from the register into same memory location. The second instruction was redundant and could be removed. Likewise, if there is a store followed by a load, it is safe to eliminate the second load instruction.

So below pattern:

        ldr     w0, [x19, #64]
        str     w0, [x19, #64]

can be optimized with:

        ldr     w0, [x19, #64]

As seen in dotnet/runtime#35613 and dotnet/runtime#35614 issues, AnalyzeAsm pointed out that this pattern occurs approximately 2570 times in 1750 methods. We are already in the process of addressing this optimization in dotnet/runtime#39222.

Replace “ldr” with “mov”

RyuJIT rarely generates code that will load two registers from same memory location, but we have seen that pattern in library methods. The second load instruction can be converted to mov instruction which is cheaper and does not need memory access.

So below pattern:

        ldr     w1, [fp,#28]
        ldr     w0, [fp,#28]

can be optimized with:

        ldr     w1, [fp,#28]
        mov     w0, w1

As seen in dotnet/runtime#35141, AnalyzeAsm pointed out that this pattern occurs approximately 540 times in 300 methods.

Loading large constants using movz/movk

Since large constants cannot be encoded in an ARM64 instruction as I have described above, we also found large number of occurrences of movz/movk pair (around 191028 of them in 4578 methods). In .NET 5, while some of these patterns are optimized by caching them as done in dotnet/runtime#39096, we are hoping to revisit other patterns and come up with a way to reduce them.

Call indirects and virtual stubs

Lastly, as I have mentioned above, 14% code size improvement in .NET libraries came from optimizing call indirects and virtual call stub in R2R code. It was possible to prioritize this from the data we obtained by using AnalyzeAsm on JIT disassembly of .NET libraries. It pointed out that the suboptimal pattern occurred approximately 615,700 times in 126,800 methods.

Techempower benchmarks

With all of the work that I described above and other work described in this blog, we made significant improvement in ARM64 performance in Techempower benchmarks. The measurements below are for Requests / Second (higher is better)

Hardware

Here are the hardware details of machines we used to run the benchmarks I have covered in this blog.

MicroBenchmarks

Our performance lab that runs microbenchmarks has following hardware configuration.

ARM64v8​
Memory:              96510MB ​
Architecture:        aarch64​
Byte Order:          Little Endian​
CPU(s):              46​
On-line CPU(s) list: 0-45​
Thread(s) per core:  1​
Core(s) per socket:  46​
Socket(s):           1​
NUMA node(s):        1​
Vendor ID:           Qualcomm​
Model:               1​
Model name:          Falkor​
Stepping:            0x0​
CPU max MHz:         2600.0000​
CPU min MHz:         600.0000​
BogoMIPS:            40.00​
L1d cache:           32K​
L1i cache:           64K​
L2 cache:            512K​
L3 cache:            58880K​
NUMA node0 CPU(s):   0-45​
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid asimdrdm

Techempower benchmarks

Our ASP.NET lab that runs techempower benchmarks has following hardware configuration.

Rack-Mount, 1U​
ThinkSystem HR330A​
1x 32-Core/3.0GHz eMAG CPU​
64GB DDR4 (8x8GB)​
1x 960GB NVMe M.2 SSD​
1x Single-Port 50GbE NIC​
2x Serial Ports​
1x 1GbE Management Port​
Ubuntu 18.04​
ARMv8​

Architecture:        aarch64​
Byte Order:          Little Endian​
CPU(s):              32​
On-line CPU(s) list: 0-31​
Thread(s) per core:  1​
Core(s) per socket:  32​
Socket(s):           1​
NUMA node(s):        1​
Vendor ID:           APM​
Model:               2​
Model name:          X-Gene​
Stepping:            0x3​
CPU max MHz:         3300.0000​
CPU min MHz:         363.9700​
BogoMIPS:            80.00​
L1d cache:           32K​
L1i cache:           32K​
L2 cache:            256K​
NUMA node0 CPU(s):   0-31

Conclusion

In .NET 5, we made great progress in improving the speed and code size for ARM64 target. Not only did we expose ARM64 intrinsics in .NET APIs, but also consumed them in our library code to optimize critical methods. With our data-driven engineering approach, we were able to prioritize high impacting work items in .NET 5. While doing performance investigation, we have also discovered several opportunities as summarized in dotnet/runtime#35853 that we plan to continue working for .NET 6. We had great partnership with @TamarChristinaArm from Arm Holdings who not only implemented some of the ARM64 hardware intrinsics, but also gave valuable suggestions and feedback to improve our code quality. We want to thank multiple contributors who made it possible to ship .NET 5 running on ARM64 target.

I would encourage you to download the latest bits of .NET 5 for ARM64 and let us know your feedback.

Happy coding on ARM64!