Dev Drive and Copy-on-Write for Developer Performance

Erik Mavrinac

Later posts in series: Dev Drive has shipped! and CoW-in-Win32 early access

At Microsoft Build 2023 the Windows team announced Dev Drive, a new evolution of the Windows ReFS filesystem retuned for developer workloads like Git and builds. This new functionality will ship later this year in the Windows 11 23H2 refresh and is available now for early testing via the Windows Insider program.

Microsoft’s One Engineering System team has been testing Dev Drive on internal Windows builds for several months. Building a large internal codebase with more than 500 C# projects, we’re seeing 14% faster full builds moving from NTFS on Windows 11 22H2 to Dev Drive.

There’s even more great news! With the Microsoft.Build.CopyOnWrite package added to the repo we’re getting an additional 14 percentage point performance improvement – a total of 28% faster – just by adding a Dev Drive volume and moving package caches and code to it.

(Methodology note: In all cases the repo directory was excluded from Defender real-time antivirus scanning, a commonly deployed configuration. Testing was not done with the new performance mode of Defender antivirus also announced at Build.)

NTFS also benefits from performance updates in the upcoming Windows release. In early testing we see a 9% improvement versus 22H2 when building with the same repo.

Copy-on-Write on ReFS and Dev Drive

Copy-on-write (CoW) linking, also known as block cloning in the Windows API documentation, avoids fully copying a file by creating a metadata reference to the original data on-disk. CoW links are like hardlinks but are safe to write to, as the filesystem lazily copies the original data into the link as needed when opened for append or random-access write. With a CoW link you save disk space and time since the link consists of a small amount of metadata and they write fast.

CoW linking is available on Windows only in the ReFS filesystem; it’s also available in iOS and Linux. Dev Drive is a special “trusted” mode for ReFS. We maintain a public repo and NuGet package for CoW for .NET.

CoW links, instead of file copies, are an excellent match for any build or tool that:

  • Makes explicit file copies to create image layouts, such as for microservice deployments.
  • Uses MSBuild or dotnet for C# builds, as the Resolve Assembly References build task copies .NET assemblies from package directories and parent projects into each new output directory.
  • Uses MSBuild and other items where MSBuild performs a copy recursively in each dependent project. (See Appendix B for ways to change this behavior.)
  • Uses caching build engines and accelerators, such as Microsoft Build Accelerator, that copy files from a local cache directory into the filesystem when a cache hit is processed.

Best Practices for CoW

All files to be CoW linked must reside within the same Dev Drive volume. This should include package caches and any other sources of files copied by the build engine or tools. Instructions for changing your machine’s environment to move package caches are detailed in Appendix A below.

Future of Dev Drive and CoW

Dev Drive and all related performance improvements will also be available in Windows Server. Internally, we are experimenting with automatic Dev Drive and package cache setup for Microsoft Dev Box images for later public rollout. We’ll also add Dev Drive to our internal CloudBuild distributed build system for faster builds and better machine utilization. After Dev Drive is released as part of the regular Windows refreshes we’ll see Dev Drive rolled out to build and release pipelines in Azure Pipelines and GitHub Actions to make it easier to benefit from performance and CoW optimizations.

Dev Drive is one of the most exciting build performance wins I’ve seen in a long time. I’m looking forward to Dev Drive appearing in the main Windows release channel later this year. Give it a try and see if you agree!

More links:

Appendix A: How to move package caches

NuGet

The NuGet global packages folder is used by dotnet, MSBuild, and Visual Studio. Create a NuGet root directory in your CoW filesystem, then set a global environment variable NUGET_PACKAGES to that path, e.g. setx /M NUGET_PACKAGES D:\packages\nuget. If you have already installed Visual Studio on your machine, move the contents of %USERPROFILE%\.nuget\packages to this directory. NuGet documentation.

NPM (NodeJS)

Create an NPM cache directory in your CoW filesystem, then set a global environment variable npm_config_cache to that path, e.g. setx /M npm_config_cache D:\packages\npm. If you have already installed NodeJS on your machine, move the contents of %AppData%\npm-cache to this directory. NPM cache, config documentation.

Yarn (NodeJS)

Create an NPM cache directory in your CoW filesystem, then set a global environment variable YARN_CACHE_FOLDER to that path, e.g. setx /M YARN_CACHE_FOLDER D:\packages\npm. Yarn documentation.

vcpkg

Create a vcpkg cache directory in your CoW filesystem, then set a global environment variable VCPKG_DEFAULT_BINARY_CACHE to that path, e.g. setx /M VCPKG_DEFAULT_BINARY_CACHE D:\packages\vcpkg. If you have already installed packages, move the contents of %LOCALAPPDATA%\vcpkg\archives or %APPDATA%\vcpkg\archives to this directory. vcpkg documentation.

Pip (Python)

Create a Pip cache directory in your CoW filesystem, then set a global environment variable PIP_CACHE_DIR to that path, e.g. setx /M PIP_CACHE_DIR D:\packages\pip. If you have already restored Pip packages and Wheels on your machine, move the contents of %LocalAppData%\pip\Cache to this directory. Pip cache documentation, StackOverflow question.

Cargo (Rust)

Create a Cargo cache directory in your CoW filesystem, then set a global environment variable CARGO_HOME to that path, e.g. setx /M CARGO_HOME D:\packages\cargo. If you have already restored Cargo packages on your machine, move the contents of %USERPROFILE%\.cargo to this directory. Cargo documentation.

Maven (Java)

Create a Maven cache directory in your CoW filesystem, then set a global environment variable MAVEN_OPTS to add a configuration setting to that path, e.g. setx /M MAVEN_OPTS “-Dmaven.repo.local=D:\packages\maven %MAVEN_OPTS%”. Move the contents of %USERPROFILE%\.m2 to this directory. StackOverflow question, Maven documentation.

Appendix B: How to disable default file copying in MSBuild

In MSBuild 17 and higher the default for and other file reference items is to copy them to dependent projects. To restore older behavior, or to otherwise optimize I/O in your build, set this in your Directory.Build.props:

<msbuildcopycontenttransitively>false</msbuildcopycontenttransitively>

This will stop transitive copying globally. You will need to fix up copying for any individual projects that are broken by the global change. Detecting breakage may not be a matter of a build break, but might be a unit test, integration test, or deployment break instead. To restore copies for these cases:

  • Individual projects that need copies from parents can set the property back to true to copy all files from parent projects, or
  • Add individual Content items as needed, or
  • Add Copy task calls in a custom target.

4 comments

Discussion is closed. Login to edit/delete existing comments.

  • Eli Black 1

    Very exciting! 🙂

    My only fear is that developers will start abusing this by also installing apps to the dev drive.

    • Erik MavrinacMicrosoft employee 1

      I’m not sure how that’s an abuse of Dev Drive. It’s a filesystem, much as NTFS is, but with optimizations for small file writes and copy-on-write capability. An installed app’s files will not be written again except in update scenarios, but otherwise the app files are just more files to read from the filesystem.

  • Bernd Eckenfels 1

    You mention java package cache relocation in this blog, but I asume Java (maven/gradel) does (not yet?) make use of the new copy-on-link ApI. Is that correct? If so are you plan to provide Java bindings and/or integrate it with Maven?

    • Erik MavrinacMicrosoft employee 2

      I personally have no such plans, and am not wired into the Maven ecosystem. I simply added information for all the package managers I could think of for completeness. My hope is that copy-on-write is added as a default into the filesystem itself so that tools do not need to be upgraded individually. Or else that Maven will make use of this in the future. We will see what happens over time as Dev Drive becomes generally available later this year and as more people start using it. Notably there is a proposal to add CoW to .NET 8.

Feedback usabilla icon