(Edited Apr 4, 2024 with release info.)

Later posts in series: Dev Drive has shipped!, CoW-in-Win32 early access, Copy-on-Write performance and debugging

At Microsoft Build 2023 the Windows team announced Dev Drive, a new evolution of the Windows ReFS filesystem retuned for developer workloads like Git and builds. This new functionality will ship later this year in the Windows 11 23H2 refresh and is available now for early testing via the Windows Insider program.

Microsoft’s One Engineering System team has been testing Dev Drive on internal Windows builds for several months. Building a large internal codebase with more than 500 C# projects, we’re seeing 14% faster full builds moving from NTFS on Windows 11 22H2 to Dev Drive.

There’s even more great news! With the Microsoft.Build.CopyOnWrite package added to the repo we’re getting an additional 14 percentage point performance improvement – a total of 28% faster – just by adding a Dev Drive volume and moving package caches and code to it.

(Methodology note: In all cases the repo directory was excluded from Defender real-time antivirus scanning, a commonly deployed configuration. Testing was not done with the new performance mode of Defender antivirus also announced at Build.)

NTFS also benefits from performance updates in the upcoming Windows release. In early testing we see a 9% improvement versus 22H2 when building with the same repo.

Copy-on-Write on ReFS and Dev Drive

Copy-on-write (CoW) linking, also known as block cloning in the Windows API documentation, avoids fully copying a file by creating a metadata reference to the original data on-disk. CoW links are like hardlinks but are safe to write to, as the filesystem lazily copies the original data into the link as needed when opened for append or random-access write. With a CoW link you save disk space and time since the link consists of a small amount of metadata and they write fast.

CoW linking is available on Windows only in the ReFS filesystem; it’s also available in iOS and Linux. Dev Drive is a special “trusted” mode for ReFS. We maintain a public repo and NuGet package for CoW for .NET.

CoW links, instead of file copies, are an excellent match for any build or tool that:

• Makes explicit file copies to create image layouts, such as for microservice deployments.
• Uses MSBuild or dotnet for C# builds, as the Resolve Assembly References build task copies .NET assemblies from package directories and parent projects into each new output directory.
• Uses MSBuild and other items where MSBuild performs a copy recursively in each dependent project. (See Appendix B for ways to change this behavior.)
• Uses caching build engines and accelerators, such as Microsoft Build Accelerator, that copy files from a local cache directory into the filesystem when a cache hit is processed.

Best Practices for CoW

All files to be CoW linked must reside within the same Dev Drive volume. This should include package caches and any other sources of files copied by the build engine or tools. Instructions for changing your machine’s environment to move package caches are detailed in Appendix A below.

Future of Dev Drive and CoW

Dev Drive and all related performance improvements will also be available in the next major version of Windows Server after Server 2022. Internally, we are experimenting with automatic Dev Drive and package cache setup for Microsoft Dev Box images for later public rollout. We’ll also add Dev Drive to our internal CloudBuild distributed build system for faster builds and better machine utilization. After Dev Drive is released as part of the regular Windows refreshes we’ll see Dev Drive rolled out to build and release pipelines in Azure Pipelines and GitHub Actions to make it easier to benefit from performance and CoW optimizations. Also see the later post that discusses CoW built into the Windows copy API.

Dev Drive is one of the most exciting build performance wins I’ve seen in a long time. I’m looking forward to Dev Drive appearing in the main Windows release channel later this year. Give it a try and see if you agree!

More links:

• Windows team blog post on Dev Drive.
• Visual Studio team blog post on Dev Drive and developer tools.

Appendix A: How to move package caches

NuGet

The NuGet global packages folder is used by dotnet, MSBuild, and Visual Studio. Create a NuGet root directory in your CoW filesystem, then set a global environment variable NUGET_PACKAGES to that path, e.g. setx /M NUGET_PACKAGES D:\packages\nuget. If you have already installed Visual Studio on your machine, move the contents of %USERPROFILE%\.nuget\packages to this directory. NuGet documentation.

NPM (NodeJS)

Create an NPM cache directory in your CoW filesystem, then set a global environment variable npm_config_cache to that path, e.g. setx /M npm_config_cache D:\packages\npm. If you have already installed NodeJS on your machine, move the contents of %AppData%\npm-cache to this directory. NPM cache, config documentation.

Yarn (NodeJS)

Create an NPM cache directory in your CoW filesystem, then set a global environment variable YARN_CACHE_FOLDER to that path, e.g. setx /M YARN_CACHE_FOLDER D:\packages\npm. Yarn documentation.

vcpkg

Create a vcpkg cache directory in your CoW filesystem, then set a global environment variable VCPKG_DEFAULT_BINARY_CACHE to that path, e.g. setx /M VCPKG_DEFAULT_BINARY_CACHE D:\packages\vcpkg. If you have already installed packages, move the contents of %LOCALAPPDATA%\vcpkg\archives or %APPDATA%\vcpkg\archives to this directory. vcpkg documentation.

Pip (Python)

Create a Pip cache directory in your CoW filesystem, then set a global environment variable PIP_CACHE_DIR to that path, e.g. setx /M PIP_CACHE_DIR D:\packages\pip. If you have already restored Pip packages and Wheels on your machine, move the contents of %LocalAppData%\pip\Cache to this directory. Pip cache documentation, StackOverflow question.

Cargo (Rust)

Create a Cargo cache directory in your CoW filesystem, then set a global environment variable CARGO_HOME to that path, e.g. setx /M CARGO_HOME D:\packages\cargo. If you have already restored Cargo packages on your machine, move the contents of %USERPROFILE%\.cargo to this directory. Cargo documentation.

Maven (Java)

Create a Maven cache directory in your CoW filesystem, then set a global environment variable MAVEN_OPTS to add a configuration setting to that path, e.g. setx /M MAVEN_OPTS “-Dmaven.repo.local=D:\packages\maven %MAVEN_OPTS%”. Move the contents of %USERPROFILE%\.m2 to this directory. StackOverflow question, Maven documentation.

Appendix B: How to disable default file copying in MSBuild

In MSBuild 17 and higher the default for and other file reference items is to copy them to dependent projects. To restore older behavior, or to otherwise optimize I/O in your build, set this in your Directory.Build.props:

<msbuildcopycontenttransitively>false</msbuildcopycontenttransitively>

This will stop transitive copying globally. You will need to fix up copying for any individual projects that are broken by the global change. Detecting breakage may not be a matter of a build break, but might be a unit test, integration test, or deployment break instead. To restore copies for these cases:

• Individual projects that need copies from parents can set the property back to true to copy all files from parent projects, or
• Add individual Content items as needed, or
• Add Copy task calls in a custom target.