Improve Parallelism in MSBuild

Avatar

Starting in Visual Studio 2019 16.3 we have been adding features to improve build parallelism. These features are still experimental, so they are off by default. When developing tools for Android, we introduced clang/gcc to the MSBuild platform. Clang/gcc relied on the parallelism model of the build system but MSBuild only parallelizes at the project level. This led to the creation of Multi-ToolTask (MTT) as a MSBuild Task. It forgoes MSBuild batching system and works around the typical single task limitations. This allows tasks to execution in parallel and engage other scheduling features not present in MSBuild. In this release, we leveraging MTT to the some of the existing Vcxproj build tasks, making existing tasks more parallel and in return improving build throughput.

Getting Started

MTT can be “opt-in” by setting the MSBuild property or environment variable UseMultiToolTask to true. Its usage should be transparent during day-to-day-developer workflow and it fully supports incremental builds in the IDE and on the command line, even when toggling MTT on and off. To set properties, you can set them as environment variables or follow the instructions in Customize your build. For the best effect, apply these properties to all projects within a solution.

Why Use MTT?

When enabled, the MTT uses its built-in scheduler, which enables some features to control its throughput. By setting property EnforceProcessCountAcrossBuilds to true, this will limit the max number of process used by MTT across multiple projects and MSBuild instances. This feature should help to combat slowdown and memory bounds brought on by over-subscription. For extra control, use the MultiProcMaxCount or CL_MPCount properties to define the max number of jobs. CL_MPCount property is set by the IDE (Tools > Options > Projects and Solutions > Maximum Concurrent C++ Compilations). By default, MultiProcMaxCount and CL_MPCount value are equal to the number of CPU logical processors.

Lastly, setting the metadata MultiToolTaskDependency on an item will create a dependency on another item in the same MTT instance. For example, in our project system, we build .cpp source files to generate PCH first and then build their consumer. In MTT, it is possible to describe this dependency and the scheduler will handle the order. With the dependency description, it allows .cpp without dependency on the PCH to run without waiting, opening more parallelism opportunities.

Performance gains will vary between sources base. Kevin wrote this blog to use xperf to measure your performence.  In this release, MTT is only coded to parallelize MIDL, CL, Clang, and FXC (hlsl).  If your project is using Custom Build Tools, then enable Parallel Custom Build Tools with a few clicks.  If there are other tools that you think could benefit, send us feedback.

Send Us Feedback

The feature is still experimental, and so we are still looking for ways to improve it. Tell us what your experience was or suggest ways to improve the system. Our focus is correctness, incrementality, and scalability. Leave your comments below or email us visualcpp@microsoft.com.

 

 

Avatar
Felix Huang

Follow    

16 comments

Leave a comment

  • Avatar
    Edward Lambert (SI)

    Hi, I’ve been waiting for this feature for so long! Unfortunately my dreams haven’t quite come true yet, as it doesn’t seem to be working for me in 16.4.3

    I’ve added (can’t put full xml as the blog swallows it)
    UseMultiToolTask: true
    EnforceProcessCountAcrossBuilds: true

    to my vcxproj files and removed the old /MP hacky system (MultiProcessorCompilation: true)

    However I am only getting one cl.exe per project so its ending up much slower than with /MP

    I tried removing EnforceProcessCountAcrossBuilds in case that was broken, but no difference, I also tried using MultiProcessorCompilation alongside it but just got the old behaviour back again of X instances of cl.exe per project.

    Any ideads what my issue could be?

    • Avatar
      Felix Huang

      Hello Edward, are you targeting the current PlatformToolset of v142? The build system will redirect to the previous build system for that specific version. Unfortunately, prior platform toolset won’t have this support.

      I found it easy to set with environment variable, either in the console or as the current user.
      set UseMultiToolTask=true
      set EnforceProcessCountAcrossBuilds=true

      global user:
      setx UseMultiToolTask true
      setx EnforceProcessCountAcrossBuilds true

      Thanks,
      Felix

      • Avatar
        Edward Lambert (SI)

        Hi Felix

        Thanks for replying, I am using v142 yes.

        Interestingly if I set it as an environment variable it does indeed work, but not if I add it to the vcxproj in place of MultiProcessorCompilation – do these settings need to go in a specific section of the project xml? Globals perhaps?

        The bad news is that for building our solution using this instead of /MP is slower

        standard – 9m:02s
        new msbuild scheduler – 11m:14s

        As far as I can ascertain the reason for this is that with the new mode you get a massive turnover in cl.exe instances where as /MP generally keeps the same instances open and with process creation on Windows being quite slow it is outweighing the benefits of reducing the oversubscription.

        Thanks,
        Ed.

        • Avatar
          Michael

          EnforceProcessCountAcrossBuilds and UseMultiToolTask are properties, so they’ll need to go in a PropertyGroup block. MultiProcessorCompilation is a customization on ClCompile items, so that’s why the substitution didn’t cause the new scheduler to come into play.

        • Avatar
          Felix Huang

          Thank you for trying it out. Yes, there is a perf downside due to spawning many more cl.exe. Especially true if compile unit is small to not justify the spawn time. I don’t expect to win very case against CL /MP, but I do hope that everyone could benefit from some form or parallelism, either that is /MP or MTT.

          That said, MIDL and FXC will get a major boost as they didn’t support parallelism prior.

          PS. I forgot the mention, you don’t have to turn MultiProcessorCompilation off. When UseMultiToolTask is enabled, it will disable /MP.

  • Andrey Kobrin
    Andrey Kobrin

    Hi. I’m using MultiToolTask mode in 16.4.4 version (8 parallel jobs).
    Yes, it works much faster for big solutions.
    But I’m getting an issue with unnecessary rebuilds.
    This is what I’m getting in MSBuild log (changed file names):

    Read Tracking Logs:
    C:\src\bin\vs2019\Release_x64\MyProject\MyProject.tlog\Microsoft.Build.CPPTasks.CL.read.1.tlog
    Outputs for C:\SRC\MYPROJECT\MYFILE.CPP|C:\SRC\MYPROJECT\MYFILE1.CPP|C:\SRC\MYPROJECT\MYFILE2.CPP:
    C:\SRC\BIN\VS2019\RELEASE_X64\MYPROJECT\MYFILE1.OBJ
    C:\SRC\BIN\VS2019\RELEASE_X64\MYPROJECT\MYFILE2.OBJ
    ..\MyFile.cpp will be compiled because it was not found in the tracking log.

    Microsoft.Build.CPPTasks.CL.write.1.tlog file actually has an entry for this source/object file:
    ^C:\SRC\MYPROJECT\MYFILE.CPP
    C:\SRC\BIN\VS2019\RELEASE_X64\MYPROJECT\MYFILE.OBJ
    But MYFILE.OBJ is not listed in “Outputs” in MSBuild log.

    So some files are getting recompiled for no reason – no source code changes.
    Project file: c:\src\MyProject\vs2019\MyProject.vcxproj:

    ClCompile Include=”..\myfile.cpp”
    ClCompile Include=”..\myfile1.cpp”
    ClCompile Include=”..\myfile2.cpp”

    Is it smth to do with relative paths?
    What could be the reason for rebuilds?
    Thanks,
    Andrey

    • Avatar
      Christian Hinkamp

      Hi. I got the exact same problem with unnecessary compilations of single files “because it was not found in the tracking log”.

      As additional information: it looks like only files are affected that have the same file name in different projects (like “project_a\init.cpp”, “project_b\init.cpp”).
      Maybe this will help to find the problem.

      Thanks,
      Christian

        • Avatar
          Felix Huang

          Hi, Thank you for trying out the feature. I have found and fixed two issues that could be related.
          1) When toggling MTT on and off, incremental failed with source paths that contains “..\”.
          2) A race condition where the rooting marker (the lines starting with ^) in TLOG was wrong.
          Both these are fixed for 16.5 Preview 3.

          Let me know if there are other issues.
          ~Felix

          • Avatar
            Daniel Boca

            HI Felix,

            Thank you very much for this feature, we’ve been dreaming about it! It was very hard to find the right balance between msbuild /m and cl /mp since our machines range from 6 cores to 48 or more and memory likewise.
            I’ve been encountered the same issue plus some .pch that are being reported as not found, though they exist.
            Is this change related to MsBuild only ? or it also affects the compiler ? Could the fix be back ported to MsBuild 16.4 branch if it’s only MsBuild ?

            Update: The .pch issue happens when we have ‘camel case’ names for the precompiled header like . If we change this to lower case, it works.

            Daniel

          • Avatar
            Felix Huang

            @Daniel Boca,
            I haven’t seen any issue with regarding pch, could you check if the dependency are correct? Check if the pch are built before any of the consumers are called. If the issue does persist, open a Developer Community ticket so I can help you directly.

            There is no plans to service 16.4, in theory, you could just could replace Microsoft.Build.CPPTasks.Common.dll from newer version.

  • Avatar
    Mihai Sebea

    This is working great! thank you!
    I wonder if it’s possible for MTT to set the dependency on other project just for the linking step so the last project can start compiling even though other projects have not finished just yet.

    • Avatar
      Felix Huang

      We call that this strategy the three Build Passes. From the beginning we have designed for it, there are a lot of corner cases that makes it complicated. One case is a solution that have Tools that generate code as part of their solution. Those Tools are also source code as part of the solution. Build Passes would need to be broken into two groups and to automate this case is difficult.

      Build Passes is possible but solutions and projects would need to be changed to accommodate for it. If you are interested, I may do a separate blog about it.
      Create a feature suggestion on Develop Community and Up Vote. That is the best way to get our attention.

  • Avatar
    Frank Heimes

    This is a very good idea; so I tried it out immediately.
    Unfortunately, I also observed the double-PCH problem. I tried disabling the properties for the two affected projects.
    But I stumbled upon another problem. I frequently get:

    c1xx (0, 0)
    c1xx(0,0): Error C1083: Cannot open compiler intermediate file: ‘obj\Release32\FooBar.pch’: Invalid argument (or: Access denied)

    There are obviously two (compiler?) instances trying to access the pch file prematurely.
    This only happens if I set the UseMultiToolTask and EnforceProcessCountAcrossBuilds properties in the projects.