vcpkg Host Dependencies for Cross-Compilation

Robert Schumacher

If you’re not familiar with our C++ library manager vcpkg, welcome! This post covers an intermediate/advanced topic, so you may want to first get an overview and try things out from the vcpkg GitHub page or some of our previous blogs.

Introduction

One of the best features of C++ is that it generates tailored, specialized code for each specific machine, enabling you to squeeze every ounce of performance per watt. It enables clean abstractions to coexist peacefully with low-level platform-specific bit twiddling. However, this comes at a price for many developers that venture beyond their own machine: you must build different binaries for your developer machine compared to your final target, be it a phone, a cloud server, or an embedded microcontroller.

For most small-to-medium sized projects, this isn’t a problem. You already have a compiler, a code editor, and a build system, which is plenty to make tons of incredible applications. However, since time immemorial some developers need even more flexibility, extensibility, and power than just the compiler; they need to generate sophisticated code at build time. Maybe it’s computing a perfect hash function over your known data set, maybe it’s a table-driven parser, or maybe it’s a bunch of foreign function interface boilerplate into your embedded scripting language. For whatever reason, you need the flexibility of C++ for your development environment in addition to the final runtime target.

In this blog post we’ll cover the newly shipped vcpkg feature designed to enable all these scenarios and more: Host Dependencies.

Simultaneous Compilation

As mentioned in the introduction, because C++ compiles all the way down to the metal, you generally can’t use the same compiler and flags to simultaneously target your final runtime and your developer machine. If you’re using a multi-targeting compiler like Clang/LLVM you’ll need at least different flags and if you’re using a single-target compiler like GCC or MSVC you’ll need an entirely different compiler.

If you’re lucky, your buildsystem has specific documentation for how to handle this case. Even then, it can sometimes be extremely subtle to ensure things are wired up correctly: Did you accidentally pass the flags for the target to the developer machine build? What if you need libraries for your code generator to run? What about code generators generating code for other code generators? It’s a tricky problem space that has ramifications on every aspect of the build environment.

Triplets

In vcpkg, we label each target universe as a separate “triplet”. For example, x64 Windows Desktop using dynamic CRT and MSVC, but building static libraries might be named x64-windows-static-md. Each library built within that universe links against other libraries from that universe, keeping everything super consistent. We include many triplet definitions in the box, but you can easily make your own to tweak compiler flags or adjust settings on a per-library basis (maybe you’d like Qt to be built dynamically, but your JSON parser built statically).

Naturally, your developer environment also matches one of these universes. By default, we pick x64-windows, x64-linux, or x64-osx as appropriate but it’s fully configurable at runtime via several methods.

Host Dependencies

Despite having a triplet matching the developer environment, vcpkg didn’t have syntax for libraries to express a dependency upon a port built for that environment. We’ve gotten very far with imperfect approaches, like dynamically attempting to consume libraries from a hardcoded set of fallback triplets, however these would always fall short of the ideal and required imperfect, copied code between different ports. These workarounds also fall completely flat in manifest mode, which is specifically designed to prevent the flaky behaviors that occur with these sorts of “dynamic” accesses. The fix is, of course, a way to naturally and directly express a requirement upon ports built for the developer environment.

Enter: Host Dependencies. Available now, ports and consumers can now declare dependencies upon other ports built against the developer environment.

The syntax to activate this is simply setting "host" to true in the manifest dependency object:

{
    "dependencies": [
        { "name": "contoso-cgen", "host": true }
    ]
}

During execution, a port can rely on all host dependencies having been installed to CURRENT_HOST_INSTALLED_DIR (the analog of CURRENT_INSTALLED_DIR) and they can get the currently configured host triplet via HOST_TRIPLET (the analog of TARGET_TRIPLET).

set(CGEN ${CURRENT_HOST_INSTALLED_DIR}/tools/contoso-cgen/cgen${VCPKG_HOST_EXECUTABLE_SUFFIX})
vcpkg_cmake_configure(
    SOURCE_PATH ${SOURCE_PATH}
    OPTIONS -DCODE_GENERATOR=${CGEN}
)

For projects that have their own code generators embedded inside, it’s perfectly valid to require yourself built for the host:

{
    "name": "protobuf",
    "dependencies": [
        { "name": "protobuf", "host": true }
    ]
}

Then, the port can determine if it is cross-building or native-building by comparing the triplets:

if(HOST_TRIPLET STREQUAL TARGET_TRIPLET)
  # Native compilation, set build flags to build and install the code generator
else()
  # Cross compilation, set build flags to consume the prebuilt code generator
  # from ${CURRENT_HOST_INSTALLED_DIR}
endif()

Give us your feedback!

Try out vcpkg by visiting our GitHub repo. We welcome your feedback on the tool and the new features in our issue tracker. 

We’ve only just begun incorporating this powerful new facility into the existing catalog, such as for Boost.Build (vcpkg/vcpkg.json at master · microsoft/vcpkg (github.com)) and Protobuf (vcpkg/vcpkg.json at master · microsoft/vcpkg (github.com)). It’s an enormous step forward for vcpkg users targeting important platforms such as iOS, Android, and Emscripten/WebAssembly.

The current up-to-date documentation on Host Dependencies can be found on our GitHub at vcpkg/host-dependencies.md at master · microsoft/vcpkg (github.com).

Robert Schumacher

Follow