The Coalition Sees 27.9X Iteration Build Improvement with Visual Studio 2019

David Li

David

Visual Studio 2019 v16.8 Preview 3.2 introduces significant build and link time improvements. In this blog post, we detail how the team in The Coalition building Gears 5 tested the compile and link times in three different versions of Visual Studio. The experiment was conducted to measure not only overall end-to-end time improvements but also link time improvements in the preview compared to the current release. Using Visual Studio 2019, the team saw 2.67x faster end-to-end build times and 27.9x faster link times compared to Visual Studio 2017. 

“The improvements are massive and will greatly improve our iteration times Aurel Cordonnier from The Coalition

 

Background

Compile times are a consistent theme of feedback from Visual Studio developers. Recently, we launched numerous build optimization efforts. One of which is working closely with our colleagues across Xbox Game Studios. Here, we seek to identify opportunities to improve our Visual Studio developer experience. The following details the build and link time improvements The Coalition saw when using Visual Studio 2019 v16.8 Preview 3.2.

 

Building Gears 5

Image Gears5 2020 Optimized 1920x1080 Clean

Methodology

The Gears 5 team compiled and linked the PC version of the game, using three different build configurations sequentially. The configs compile and link a very large code base into one monolithic executable. To produce consistent results, the team chose not to use /debug:fastlink with the build machines since the machines need to publish symbols.

To measure link times, the wall clock execution time of this compilation experiment starts before link is called and ends after linking the final executable. Similarly, the end-to-end times were read from the build system output (total execution time). Then, the link time from linker’s verbose output were subtracted from the total execution time to get the compile times.  The results are an average of five tests conducted for each configuration to achieve consistency with a low variance.

 

Results

The results of the experiment showed both compile and link times significantly improved in Visual Studio 2019. The compilation time for all three configurations has been improved by a factor of 2.15. In the ship configuration, Gears 5 took 41 minutes (2461.92 seconds) to compile in Visual Studio 2017. In comparison, the compilation took only 18.76 minutes (1125.61 seconds) in Visual Studio 2019 16.8 Preview 3.2. On average, the link times improved 27.9X. The ship config took only 29.57 seconds compared to 11.28 minutes (677.07 seconds) previously. Overall, the end-to-end compilation time decreased by a factor of 2.67X.

Image gears5 linktimes

Image gears5 end2endtimes

 

Compile Time (seconds)devtestship
15.92779.802605.972461.92
16.7.51321.471155.241089.27
16.8.0 p3.21338.171193.481125.61
x times faster2.082.182.19
average x times faster2.15

 

Link Time (seconds)devtestship
15.9751.93682.06677.07
16.7.566.7662.8659.00
16.8.0 p3.226.7226.2522.90
x times faster28.1425.9829.57
average x times faster27.90

 

End-to-end (seconds)devtestship
15.93531.733288.033138.99
16.7.51388.231218.101148.27
16.8.0 p3.21364.891219.731148.51
x times faster2.592.702.73
average x times faster2.67

 

Analysis

The C++ team made large steps towards Linker performance in Visual Studio version 16.0 and 16.2 with two significant changes to the linker that help speed up the generation of Program Database (PDB) files. Type pruning in the backend removes unreferenced type information. By using a fast hash function to identify identical types, type merging sped up. Please see details in this blog post.

In version 16.6, changes were made to algorithms inside the PDB and Debug Interface Access components. Next, optimization efforts were made in version 16.7 to speed up the worst case Incremental Linking time. Later, multi-threading the generation of PDB files further improved performance in version 16.8. Please stay tuned for future blogs that dive deeper into these improvements.

 

Developer Conversation

Image G5 Kait and Marcus amulet

Aurel from The Coalition spoke to us about how improved build times have affected the studio.

What prompted the move to Visual Studio 2019?

So, we’re not actually moving Gears 5 to 2019. We did the work to get Visual Studio 2019 going on PC for the purpose of comparing build times and providing feedback. Gears 5 shipped over a year ago and the game has been very carefully tuned; therefore, we don’t want to take the risk of changing the toolchain as we have limited testing capacity and very slight optimization changes could change the balance of the game in very subtle ways. In addition, the XDK does not support 2019 so we would need to support two toolchains which is something we don’t have the capacity for.

The results of the experiment reinforced our desire to stay current with new versions of Visual Studio for our future projects.

What are some immediate benefits?

We are using Visual Studio 2019 on our next (unannounced) project and the benefits are many: faster compile/link times, more responsive UI, much more stable debugging.

What are some potential long-term benefits?

Developer iteration is key to our success, any time we can shave a few seconds off our iteration loop, it means developers have more time to write quality code, fix bugs etc. Faster build times on the build farm means our testing team can get a build sooner, can find more bugs etc.

In the end, we will end up with a better game for our fans.

How might this change the workflow patterns of the studio?

With Visual Studio 2017 (and 2019 prior to 16.8), we had to choose between fast link times or debugging stability (by using /debug:fastlink or not).

With 16.8, we can have both as link times without fastlink are very fast.

Upgrade Visual Studio

Download the latest Visual Studio 2019 Preview and experience faster end-to-end compile times for your own projects.

How will these improvements impact your day-to-day workflow? Talk to us in the comments below, on Twitter (@VisualC), or via email at visualcpp@microsoft.com

4 comments

Leave a comment

  • Zachary Turner
    Zachary Turner

    I am seeing almost no measurable impact on build times using the latest preview. Are there certain compiler options or flags that will exhibit these improvements more than others?

    My baseline version is 16.6.30309.148 where my builds take about 92 seconds, and have compared this against 16.9.30709.64 where my builds take about 90 seconds.

    This is also for a large gaming platform. Do you have any suggestions for things I might be missing or any other theories as to why I’m not seeing any improvements? I’m using /Od, /DEBUG:FULL, and /Z7

    • Gratian Lup
      Gratian LupMicrosoft employee

      Hi Zachary,

      There are no extra flags needed. Those 90 seconds, are they just the time to do the linking, or a larger part of a build?
      If it’s a build, is that a clean build or changing one cpp file for example?
      I assume the build machine is already decent, 4+ cores with enough RAM and an SSD, right?

      It would be helpful to get either a vcperf (Build insights) trace, but a plain xperf trace done with “xperf.exe -on Latency -stackwalk profile” should also be good. If you want to share that privately, you can email me at gratilup@microsoft.com

      Thanks,
      Gratian

      • Zachary Turner
        Zachary Turner

        This is part of a larger build using Ninja and includes compilation + linking of the entire application and all source files within. Also I should mention that all numbers are clean builds. One thing I found is that If I change Z7 to Zi I do see a difference between 16.6 and 16.9, but it’s still slower than if I use Z7. The times I measured were:

        /Z7 (VS 16.6): 92 seconds
        /Z7 (VS 16.9): 90 seconds
        /Zi (VS 16.6): 132 seconds
        /Zi (VS 16.9) 121 seconds

        Is it possible that most of the optimizations are in mspdbsrv and so I wouldn’t see them when using /Z7?

        If I were to send you a trace, would you be interested only in the trace using 16.9, or would the 16.6 trace be useful as well?