Introduction

The C++ team at Visual Studio has delivered substantial build and link time improvements throughout Visual Studio 2019. This blog is Part 2 of a series of blogs showcasing real-world results of our efforts. See how the Gears 5 team benefited from iteration build time improvements in Part 1.

In this blog, the Forza Horizon 4 team shares tests results of compile and link times in three different versions of Visual Studio. Link times are now 18.15X faster than in Visual Studio 2017. Likewise, Turn 10 Studios saw a 4.95X improvement in link time. The decrease in build time enabled Playground Games to switch from /debug:fastlink to/debug:full. Analyzing the results, we can see not only massive iteration build time wins but also applicable real-world quality of life improvements.

“Using full linking removes the debugger stalls associated with fastlink and allows our engineering team to focus on debugging the current build rather than waiting for the development environment to become responsive. Coupled with the overall improvements to build and link times, this leads to less workflow interruption and productivity improvements across the team.” – Andrew Sage from Playground Games

Building Forza Horizon 4

Methodology

The Forza Horizon 4 team at Playground Games measured build times using Incredibuild’s build monitor. The monitor shows wall clock time for each stage of the process. Compile time is measured at the start of the first file compile to immediately before linking. Link time is measured from start of the link step to the end of link step. Total time, measured by the start-to-end Incredibuild process, may be longer than compile and link times combined since some startup and post link steps are included in the measurement.

Tests were conducted during quiet times at the studio to ensure parallel compiles were not affected by network load. To further reduce noise in data, tests were averaged over multiple runs, and were started after all hard drive disk activity had finished and Intellisense had finished parsing.

Results

Version 15.9.4 /debug:fastlink vs 16.8 /debug:full

Comparing /debug:fastlink in Visual Studio 2017 and /debug:full in Visual Studio 2019 demonstrated real-world quality of life improvements. At first sight, total time has improved by a factor of 1.26; however, considerable time of 30-45 seconds per debug session was saved by switching to /debug:full, resulting in large productivity gains. The engineers at Playground Games have made the switch to /debug:full in order to both enjoy shorter build times and take advantage of the faster debugging experience afforded by /debug:full.

 

Total Time (minutes)	Full rebuild	Single file change
VS2017 15.9.4 /debug:fastlink	4:22	1:39
VS2019 16.8    /debug:full	3:40	1:15
x times faster	1.19	1.32
average x times faster	1.26

Version 15.9.4 vs 16.8 using /debug:full

In the second test, a comparison of builds using /debug:full was done over 3 versions of Visual Studio. The compilation time has been improved by a factor of 1.52. When doing a full rebuild, the compile time decreased from 4 minutes 39 seconds in Visual Studio 2017 to 3 minutes 5 seconds in Visual Studio 2019 16.8. For a single file change, a 19 second reduction was observed. On average, the link times improved by a factor of 18.15X. The full rebuild link time in Visual Studio 2019 16.8 decreased by 10 minutes 18 seconds, a monumental improvement.

 

Compile Time (minutes)	Full rebuild	Single file change
VS2017 15.9.4 /debug:full	4:39	0:55
VS2019 16.7.2 /debug:full	3:31	0:37
VS2019 16.8    /debug:full	3:05	0:36
x times faster	1.51	1.53
average x times faster	1.52

 

 

Link Time (minutes)	Full rebuild	Single file change
VS2017 15.9.4 /debug:full	10:53	11:28
VS2019 16.7.2 /debug:full	1:23	1:39
VS2019 16.8    /debug:full	0:35	0:39
x times faster	18.66	17.64
average x times faster	18.15

 

Total Time (minutes)	Full rebuild	Single file change
VS2017 15.9.4 /debug:full	15:32	12:23
VS2019 16.7.2 /debug:full	4:54	3:40
VS2019 16.8    /debug:full	3:40	1:15
x times faster	4.24	9.91
average x times faster	7.07

 

Building Forza Motorsport

Methodology

At Turn 10 Studios, the Forza Motorsport team used MSBuild’s Structured Log Viewer to obtain build and compile times. Each of the times below are the result of an average over 4 runs. CL refers to cl.exe (MSVC Compiler) and Lib refers to lib.exe (Microsoft Library Manager). While Full Build indicates total time, Debug, CL, and Lib times were chosen to be showcased; each time was averaged individually.

Results

Version 15.9 vs 16.7 Profile Build with /debug:fastlink

When doing a full Profile build with /debug:fastlink, we can see a moderate improvement in build time while link time has improved by a factor of 1.96. Lib time has been decreased by 9.34X.

 

Time (seconds)	Full Build (Profile)	Debug Fastlink	CL	Lib
VS2017 15.9	285.5	18	38.9	43.9
VS2019 16.7	249.75	9.2	40.5	4.7
x times faster	1.14	1.96	0.96	9.34

Version 15.9 vs 16.7 Profile Build with /debug:fulllink

Using /debug:fulllink, we can see much more favorable numbers. Full build time is now 1.71X faster. Similarly, link time has been improved by a factor of 4.95. Lib time was also improved by 10.86X.

 

Time (seconds)	Full Build (Profile)	Debug Fulllink	CL	Lib
VS2017 15.9	431	172	34	38
VS2019 16.7	251.75	34.75	34.75	3.5
x times faster	1.71	4.95	0.98	10.86

Version 15.9 vs 16.7 Release Build

For the Release Build, we see similar improvements as above. Build time decreased by 1.81X and link time improved by a factor of 2.44.

 

Time (seconds)	Full Build (Release)	Link	CL	Lib
VS2017 15.9	664	479	30	34.5
VS2019 16.7	367	196	29	29
x times faster	1.81	2.44	1.03	1.19

Analysis

The significant build iteration time improvement is a direct result of optimization efforts by the C++ Team. Linker performance was improved in version 16.0 and 16.2 by speeding up Program Database (PDB) file generation and type merging. Algorithmic changes in 16.6 and worse case Incremental Linking improvements in 16.7 increased optimization of Linker performance.  In version 16.8, multi-threading generation of PDB files further improved performance. Stay tuned for a technical blogpost detailing 16.8 improvements.

Developer Conversation

We spoke with Andrew Sage from Playground Games (Forza Horizon 4) and Dan Tunnell from Turn 10 Studios (Forza Motorsport) for some insights into how the improvements have affected their studios.

Is the studio currently using Visual Studio 2019?

“We have a mix of VS2017 and VS2019, depending on setup. Most engineers have moved to VS2019 for the main project, but we still have some tools projects that haven’t been updated yet and are still using 2017.” – Andrew

 

“We just moved the whole studio to Visual Studio 2019 in the last week of January 2021.  We’ve had one of our development branches running it since before the holidays but it just reached broad adoption a couple of days ago.” – Dan

 

What were some immediate benefits that you saw from faster link times?

“Reducing iteration time to build and test changes allows us to be more productive, especially as gains add up over the course of all of the engineers multiple times per day. The main benefit was the removal of stalls when debugging with fast link when switching to full link, as these would add 30-45 seconds per debug session, which can be repeated multiple times per build.” – Andrew

 

“The biggest benefit I see is improved iteration time.  Even if we only shave 30 seconds off link time, multiplying that by the number of devs in the studio and the number of times they compile each day becomes a large number really quick.” – Dan

 

What are some potential long-term benefits?

“Longer term, we will also be moving all our automated build servers over to Visual Studio 2019 to reduce our build validation and test turnaround times. This means a reduction in time from checking in code to knowing that the build has succeeded. Also, there will be less changes per build as these builds run continuously. Moving all our tools projects over to Visual Studio 2019 in the future would lead to a further productivity benefit for the tools team, and their ability to spend more time supporting our content creators.” – Andrew

 

“One of the longer-term benefits is reduced build times on our server farm.  If someone checks in code that breaks the build, the faster we catch it the less downtime we experience.  It also gives us faster tooling iteration since the time between a developer checking in new tooling code and the build system checking in new tools binaries is reduced.  Additionally, there’s reduced time for “validation” builds.  Prior to checking in, each developer builds a representative sample of the game binaries to ensure that they haven’t broken anything.  This slows iteration (I’m sure you can see a pattern emerging – iteration time is king) and anything we can do to speed that up is extremely valuable.” – Dan

 

How might the link time improvements change the workflow patterns of the studio?

“Besides the improvement in focusing on fixing issues rather than waiting for builds to complete across the team, I think we will be able to focus on other productivity gains which then become a larger overall proportion of our processes. For instance, reducing dependencies in our codebase can improve build times even with the faster toolset and distributed build systems, and we will continue to focus our efforts in this area in the future. Also, using full link allows for less mental ‘task switching’ overhead caused by the stalls, as often when someone hits a stall they might swap to another task for a bit and then back in later, which can cause workflow to become interrupted.” – Andrew

 

“If I were to guess, I’d say that we’ll see less build breaks.  Validation builds take a significant amount of time to complete.  Even though they’re required, sometimes developers will omit that step if they’re in a hurry and feel confident that their fix is trivial enough to be safe.  The faster the build times are the lower that barrier to entry is and the more likely people will perform that step even for changes they consider to be trivial and/or safe.  I also think we might see less code changes between compilations.  The longer compilation and linking takes the more code a developer will add before they hit the button to compile.  If compilation and linking is almost instantaneous, most developers will use that as a quick sanity check to verify that the method/function/class/etc. that they just added compiles before moving on.” – Dan

Upgrade Visual Studio

Download the latest Visual Studio 2019 Preview and experience faster end-to-end build iteration times for your own projects.

How will these improvements impact your day-to-day workflow? Talk to us in the comments below, on Twitter (@VisualC), or via email at visualcpp@microsoft.com