Visual Studio 2019 v16.8 Preview 3.2 introduces significant build and link time improvements. In this blog post, we detail how the team in The Coalition building Gears 5 tested the compile and link times in three different versions of Visual Studio. The experiment was conducted to measure not only overall end-to-end time improvements but also link time improvements in the preview compared to the current release. Using Visual Studio 2019, the team saw 2.67x faster end-to-end build times and 27.9x faster link times compared to Visual Studio 2017.
“The improvements are massive and will greatly improve our iteration times” –Aurel Cordonnier from The Coalition
Background
Compile times are a consistent theme of feedback from Visual Studio developers. Recently, we launched numerous build optimization efforts. One of which is working closely with our colleagues across Xbox Game Studios. Here, we seek to identify opportunities to improve our Visual Studio developer experience. The following details the build and link time improvements The Coalition saw when using Visual Studio 2019 v16.8 Preview 3.2.
Building Gears 5
Methodology
The Gears 5 team compiled and linked the PC version of the game, using three different build configurations sequentially. The configs compile and link a very large code base into one monolithic executable. To produce consistent results, the team chose not to use /debug:fastlink with the build machines since the machines need to publish symbols.
To measure link times, the wall clock execution time of this compilation experiment starts before link is called and ends after linking the final executable. Similarly, the end-to-end times were read from the build system output (total execution time). Then, the link time from linker’s verbose output were subtracted from the total execution time to get the compile times. The results are an average of five tests conducted for each configuration to achieve consistency with a low variance.
Results
The results of the experiment showed both compile and link times significantly improved in Visual Studio 2019. The compilation time for all three configurations has been improved by a factor of 2.15. In the ship configuration, Gears 5 took 41 minutes (2461.92 seconds) to compile in Visual Studio 2017. In comparison, the compilation took only 18.76 minutes (1125.61 seconds) in Visual Studio 2019 16.8 Preview 3.2. On average, the link times improved 27.9X. The ship config took only 29.57 seconds compared to 11.28 minutes (677.07 seconds) previously. Overall, the end-to-end compilation time decreased by a factor of 2.67X.
| Compile Time (seconds) | dev | test | ship |
| 15.9 | 2779.80 | 2605.97 | 2461.92 |
| 16.7.5 | 1321.47 | 1155.24 | 1089.27 |
| 16.8.0 p3.2 | 1338.17 | 1193.48 | 1125.61 |
| x times faster | 2.08 | 2.18 | 2.19 |
| average x times faster | 2.15 |
| Link Time (seconds) | dev | test | ship |
| 15.9 | 751.93 | 682.06 | 677.07 |
| 16.7.5 | 66.76 | 62.86 | 59.00 |
| 16.8.0 p3.2 | 26.72 | 26.25 | 22.90 |
| x times faster | 28.14 | 25.98 | 29.57 |
| average x times faster | 27.90 |
| End-to-end (seconds) | dev | test | ship |
| 15.9 | 3531.73 | 3288.03 | 3138.99 |
| 16.7.5 | 1388.23 | 1218.10 | 1148.27 |
| 16.8.0 p3.2 | 1364.89 | 1219.73 | 1148.51 |
| x times faster | 2.59 | 2.70 | 2.73 |
| average x times faster | 2.67 |
Analysis
The C++ team made large steps towards Linker performance in Visual Studio version 16.0 and 16.2 with two significant changes to the linker that help speed up the generation of Program Database (PDB) files. Type pruning in the backend removes unreferenced type information. By using a fast hash function to identify identical types, type merging sped up. Please see details in this blog post.
In version 16.6, changes were made to algorithms inside the PDB and Debug Interface Access components. Next, optimization efforts were made in version 16.7 to speed up the worst case Incremental Linking time. Later, multi-threading the generation of PDB files further improved performance in version 16.8. Please stay tuned for future blogs that dive deeper into these improvements.
Developer Conversation
Aurel from The Coalition spoke to us about how improved build times have affected the studio.
What prompted the move to Visual Studio 2019?
So, we’re not actually moving Gears 5 to 2019. We did the work to get Visual Studio 2019 going on PC for the purpose of comparing build times and providing feedback. Gears 5 shipped over a year ago and the game has been very carefully tuned; therefore, we don’t want to take the risk of changing the toolchain as we have limited testing capacity and very slight optimization changes could change the balance of the game in very subtle ways. In addition, the XDK does not support 2019 so we would need to support two toolchains which is something we don’t have the capacity for.
The results of the experiment reinforced our desire to stay current with new versions of Visual Studio for our future projects.
What are some immediate benefits?
We are using Visual Studio 2019 on our next (unannounced) project and the benefits are many: faster compile/link times, more responsive UI, much more stable debugging.
What are some potential long-term benefits?
Developer iteration is key to our success, any time we can shave a few seconds off our iteration loop, it means developers have more time to write quality code, fix bugs etc. Faster build times on the build farm means our testing team can get a build sooner, can find more bugs etc.
In the end, we will end up with a better game for our fans.
How might this change the workflow patterns of the studio?
With Visual Studio 2017 (and 2019 prior to 16.8), we had to choose between fast link times or debugging stability (by using /debug:fastlink or not).
With 16.8, we can have both as link times without fastlink are very fast.
Upgrade Visual Studio
Download the latest Visual Studio 2019 Preview and experience faster end-to-end compile times for your own projects.
How will these improvements impact your day-to-day workflow? Talk to us in the comments below, on Twitter (@VisualC), or via email at visualcpp@microsoft.com
I am seeing almost no measurable impact on build times using the latest preview. Are there certain compiler options or flags that will exhibit these improvements more than others?
My baseline version is 16.6.30309.148 where my builds take about 92 seconds, and have compared this against 16.9.30709.64 where my builds take about 90 seconds.
This is also for a large gaming platform. Do you have any suggestions for things I might be missing or any...
Hi Zachary,
There are no extra flags needed. Those 90 seconds, are they just the time to do the linking, or a larger part of a build?
If it's a build, is that a clean build or changing one cpp file for example?
I assume the build machine is already decent, 4+ cores with enough RAM and an SSD, right?
It would be helpful to get either a vcperf (Build insights) trace, but a plain xperf trace...
This is part of a larger build using Ninja and includes compilation + linking of the entire application and all source files within. Also I should mention that all numbers are clean builds. One thing I found is that If I change Z7 to Zi I do see a difference between 16.6 and 16.9, but it's still slower than if I use Z7. The times I measured were:
/Z7 (VS 16.6): 92 secondsRead more
No, mspdbsrv doesn't use this PDB multi-threading, it's just at link time when producing the final PDB. Gears of Wars uses /Z7 for ex. A trace for both 16.6 and 16.9 would be great, yes. Build insights will have the right info, in case you didn't use it before getting started is here:
https://docs.microsoft.com/en-us/cpp/build-insights/get-started-with-cpp-build-insights?view=msvc-160
I suspect the link itself is only a couple of seconds and any speedup would have a small impact on the entire...