January 29th, 2020

Improve Parallelism in MSBuild

Starting in Visual Studio 2019 16.3 we have been adding features to improve build parallelism. These features are still experimental, so they are off by default. When developing tools for Android, we introduced clang/gcc to the MSBuild platform. Clang/gcc relied on the parallelism model of the build system but MSBuild only parallelizes at the project level. This led to the creation of Multi-ToolTask (MTT) as a MSBuild Task. It forgoes MSBuild batching system and works around the typical single task limitations. This allows tasks to execution in parallel and engage other scheduling features not present in MSBuild. In this release, we leveraging MTT to the some of the existing Vcxproj build tasks, making existing tasks more parallel and in return improving build throughput.

Getting Started

MTT can be “opt-in” by setting the MSBuild property or environment variable UseMultiToolTask to true. Its usage should be transparent during day-to-day-developer workflow and it fully supports incremental builds in the IDE and on the command line, even when toggling MTT on and off. To set properties, you can set them as environment variables or follow the instructions in Customize your build. For the best effect, apply these properties to all projects within a solution.

Why Use MTT?

When enabled, the MTT uses its built-in scheduler, which enables some features to control its throughput. By setting property EnforceProcessCountAcrossBuilds to true, this will limit the max number of process used by MTT across multiple projects and MSBuild instances. This feature should help to combat slowdown and memory bounds brought on by over-subscription. For extra control, use the MultiProcMaxCount or CL_MPCount properties to define the max number of jobs. CL_MPCount property is set by the IDE (Tools > Options > Projects and Solutions > Maximum Concurrent C++ Compilations). By default, MultiProcMaxCount and CL_MPCount value are equal to the number of CPU logical processors.

Lastly, setting the metadata MultiToolTaskDependency on an item will create a dependency on another item in the same MTT instance. For example, in our project system, we build .cpp source files to generate PCH first and then build their consumer. In MTT, it is possible to describe this dependency and the scheduler will handle the order. With the dependency description, it allows .cpp without dependency on the PCH to run without waiting, opening more parallelism opportunities.

Performance gains will vary between sources base. Kevin wrote this blog to use xperf to measure your performence.  In this release, MTT is only coded to parallelize MIDL, CL, Clang, and FXC (hlsl).  If your project is using Custom Build Tools, then enable Parallel Custom Build Tools with a few clicks.  If there are other tools that you think could benefit, send us feedback.

Send Us Feedback

The feature is still experimental, and so we are still looking for ways to improve it. Tell us what your experience was or suggest ways to improve the system. Our focus is correctness, incrementality, and scalability. Leave your comments below or email us visualcpp@microsoft.com.

 

 

Category
Announcement

Author

17 comments

Discussion is closed. Login to edit/delete existing comments.

  • Frank Heimes

    This is a very good idea; so I tried it out immediately.
    Unfortunately, I also observed the double-PCH problem. I tried disabling the properties for the two affected projects.
    But I stumbled upon another problem. I frequently get:

    c1xx (0, 0)
    c1xx(0,0): Error C1083: Cannot open compiler intermediate file: 'obj\Release32\FooBar.pch': Invalid argument (or: Access denied)

    There are obviously two (compiler?) instances trying to access the pch file prematurely.
    This only happens if I set the...

    Read more
    • Felix HuangMicrosoft employee Author

      Hello, Frank. I have tested with multiple PCH, but I have come to learn, there are multiple ways to set them up. Could you please open a Developer Community ticket and share a sample project?
      Thanks.
      Felix

  • Stefan Sieber

    Does not work for projects with multiple PCHs, no processes (cl.exe) are spawned by the associated MSBuild instance.

    • Felix HuangMicrosoft employee Author

      Thanks I can repro the issue. I will let you know when I have a solution.

  • Mihai Sebea

    This is working great! thank you!
    I wonder if it’s possible for MTT to set the dependency on other project just for the linking step so the last project can start compiling even though other projects have not finished just yet.

    • Felix HuangMicrosoft employee Author

      We call that this strategy the three Build Passes. From the beginning we have designed for it, there are a lot of corner cases that makes it complicated. One case is a solution that have Tools that generate code as part of their solution. Those Tools are also source code as part of the solution. Build Passes would need to be broken into two groups and to automate this case is...

      Read more
  • Andrey Kobrin

    Hi. I'm using MultiToolTask mode in 16.4.4 version (8 parallel jobs).
    Yes, it works much faster for big solutions.
    But I'm getting an issue with unnecessary rebuilds.
    This is what I'm getting in MSBuild log (changed file names):

    Read Tracking Logs:
    C:\src\bin\vs2019\Release_x64\MyProject\MyProject.tlog\Microsoft.Build.CPPTasks.CL.read.1.tlog
    Outputs for C:\SRC\MYPROJECT\MYFILE.CPP|C:\SRC\MYPROJECT\MYFILE1.CPP|C:\SRC\MYPROJECT\MYFILE2.CPP:
    C:\SRC\BIN\VS2019\RELEASE_X64\MYPROJECT\MYFILE1.OBJ
    ...

    Read more
    • Christian Hinkamp

      Hi. I got the exact same problem with unnecessary compilations of single files “because it was not found in the tracking log”.

      As additional information: it looks like only files are affected that have the same file name in different projects (like “project_a\init.cpp”, “project_b\init.cpp”).
      Maybe this will help to find the problem.

      Thanks,
      Christian

      • Christian Hinkamp

        Addendum: the file “project_b\init.cpp” doesn’t even exist – it only appears as an entry in the intermediate log file from project_a…
        So my guess was wrong – but maybe it will help anyway…

      • Felix HuangMicrosoft employee Author

        Hi, Thank you for trying out the feature. I have found and fixed two issues that could be related.
        1) When toggling MTT on and off, incremental failed with source paths that contains “..\”.
        2) A race condition where the rooting marker (the lines starting with ^) in TLOG was wrong.
        Both these are fixed for 16.5 Preview 3.

        Let me know if there are other issues.
        ~Felix

      • Felix HuangMicrosoft employee Author

        @Daniel Boca,
        I haven’t seen any issue with regarding pch, could you check if the dependency are correct? Check if the pch are built before any of the consumers are called. If the issue does persist, open a Developer Community ticket so I can help you directly.

        There is no plans to service 16.4, in theory, you could just could replace Microsoft.Build.CPPTasks.Common.dll from newer version.

      • Daniel BocaMicrosoft employee

        HI Felix,

        Thank you very much for this feature, we've been dreaming about it! It was very hard to find the right balance between msbuild /m and cl /mp since our machines range from 6 cores to 48 or more and memory likewise.
        I've been encountered the same issue plus some .pch that are being reported as not found, though they exist.
        Is this change related to MsBuild only ? or it also affects the compiler...

        Read more
  • Edward Lambert (SI)

    Hi, I've been waiting for this feature for so long! Unfortunately my dreams haven't quite come true yet, as it doesn't seem to be working for me in 16.4.3

    I've added (can't put full xml as the blog swallows it)
    UseMultiToolTask: true
    EnforceProcessCountAcrossBuilds: true

    to my vcxproj files and removed the old /MP hacky system (MultiProcessorCompilation: true)

    However I am only getting one cl.exe per project so its ending up much slower than with /MP

    I tried removing EnforceProcessCountAcrossBuilds...

    Read more
    • Felix HuangMicrosoft employee Author

      Hello Edward, are you targeting the current PlatformToolset of v142? The build system will redirect to the previous build system for that specific version. Unfortunately, prior platform toolset won’t have this support.

      I found it easy to set with environment variable, either in the console or as the current user.
      set UseMultiToolTask=true
      set EnforceProcessCountAcrossBuilds=true

      global user:
      setx UseMultiToolTask true
      setx EnforceProcessCountAcrossBuilds true

      Thanks,
      Felix

      • Edward Lambert (SI)

        Hi Felix

        Thanks for replying, I am using v142 yes.

        Interestingly if I set it as an environment variable it does indeed work, but not if I add it to the vcxproj in place of MultiProcessorCompilation - do these settings need to go in a specific section of the project xml? Globals perhaps?

        The bad news is that for building our solution using this instead of /MP is slower

        standard - 9m:02s
        new msbuild scheduler - 11m:14s

        As far as...

        Read more
      • Felix HuangMicrosoft employee Author

        Thank you for trying it out. Yes, there is a perf downside due to spawning many more cl.exe. Especially true if compile unit is small to not justify the spawn time. I don't expect to win very case against CL /MP, but I do hope that everyone could benefit from some form or parallelism, either that is /MP or MTT.

        That said, MIDL and FXC will get a major boost as they didn't...

        Read more
      • Michael

        EnforceProcessCountAcrossBuilds and UseMultiToolTask are properties, so they’ll need to go in a PropertyGroup block. MultiProcessorCompilation is a customization on ClCompile items, so that’s why the substitution didn’t cause the new scheduler to come into play.