Take control of your vcpkg dependencies with versioning support

Augustin Popa

Special thanks to Victor Romero for putting together the content for this blog post.

We have an exciting new feature to announce in vcpkg: the long-awaited and highly requested package versioning! This feature makes it possible to install specific versions of dependencies and control installed versions over time. In order to use this feature, a vcpkg.json manifest file must be present in your repo to declare dependencies. Versioning is not currently available for libraries installed via the command line (i.e. vcpkg install library_name commands). The versioning feature is completely optional – you can choose not to specify library versions, and vcpkg will pick the most appropriate set of compatible versions for your dependencies from its baseline catalog. For now, we are considering this feature experimental. Please give us your feedback and let us know how we can improve.

 

Announcing package versioning support

For the past year we have been focusing our efforts on implementing highly requested features in vcpkg that will help our users in a wide variety of scenarios. Two such features are manifests and binary caching, which we announced some time ago, and users have been successfully using them in their projects since then. Today, we are announcing support for another anticipated feature: Package versioning. With this feature users will be able to:

  • Declare minimum version constraints on dependencies.
  • Freeze dependencies at specific versions.
  • Conveniently upgrade all declared dependencies at once via baselines.
  • Get reproducible builds independent from the current state of the vcpkg ports registry.

 

Getting started with versions

To enable versioning, you must turn on the “versions” feature flag. There are several ways to do this:

  • Setting the VCPKG_FEATURE_FLAGS environment variable, example in Windows 10:Image versioning feature flag
  • Setting the VCPKG_FEATURE_FLAGS variable before invoking vcpkg in the command line (example in PowerShell): $env:VCPKG_FEATURE_FLAGS="versions" vcpkg install
  • Passing the feature flags in the command line for vcpkg (example in PowerShell): vcpkg --feature-flags="versions" install

In the example below, we will be using Visual Studio Code to create a simple CMake project that automatically reads a vcpkg manifest file and installs dependencies. You can do this in Visual Studio as well. For information on how to set up Visual Studio for use with vcpkg with manifests (for MSBuild or CMake), see vcpkg: Accelerate your team development environment with binary caching and manifests.

Example #1: Simple versioning

Begin by creating a folder with the following files:

vcpkg.json

{
    "name": "versions-test",
    "version": "1.0.0",
    "dependencies": [
        {
            "name": "fmt",
            "version>=": "7.1.3"
        },
        "zlib"
    ],
    "builtin-baseline": "b60f003ccf5fe8613d029f49f835c8929a66eb61"
}

vcpkg has new version declaration properties when you use manifests. Previously, you could only declare versions for your projects using the “version-string” property. Now that versioning has come around, vcpkg is aware of some new versioning schemes.

Version scheme Description
version Dot-separated numerals: 1.0.0
version-semver Compliant semantic versions: 1.2.0, 1.2.1-rc
version-date Dates in YYYY-MM-DD format: 2021-01-01
version-string Arbitrary strings: vista, xp

 

The selected versioning scheme has consequences in what vcpkg will allow as a valid version string and the rules for ordering versions. You can read more about versioning schemes in our documentation.

Second, we use the “version>=” property to declare a minimum version constraint on fmt. Notice that we also declare a dependency on zlib without any version constraint.

And lastly, we declare a “builtin-baseline”, the value of which is a commit SHA from the vcpkg repository.

In our example, vcpkg will look inside commit b60f003ccf5fe8613d029f49f835c8929a66eb61 and find what the latest versions of fmt and zlib at that point in time were:

  • fmt 7.1.3
  • zlib 1.2.11#9 (the #9 suffix indicates that this is the 9th version of this library build recipe)

The set of libraries and versions listed above can be described as the baseline versions for the dependencies used in this project. Baseline versions get added as additional minimum version constraints when resolving package versions.

main.cpp

#include <fmt/core.h>
#include <zlib.h>

int main()
{
    fmt::print("fmt version is {}\n"
               "zlib version is {}\n",
               FMT_VERSION, ZLIB_VERSION);
    return 0;
}

[Above] This is a simple single file program to test that dependencies are correctly installed.

CMakeLists.txt

cmake_minimum_required(VERSION 3.18)

set(VCPKG_FEATURE_FLAGS "versions")
project(versions-test CXX)

add_executable(main main.cpp)

find_package(ZLIB REQUIRED)
find_package(fmt CONFIG REQUIRED)
target_link_libraries(main PRIVATE ZLIB::ZLIB fmt::fmt)

[Above] To use vcpkg manifests with a CMake project, it is necessary to add find_package and target_link_libraries functions to identify your dependencies in CMakeLists.txt. This experience is the same whether a package manager like vcpkg is being used or not. Including these lines makes it possible for dependencies to be included in builds.

.vscode/settings.json

{
    "cmake.configureSettings": {
        "CMAKE_TOOLCHAIN_FILE": "D:/vcpkg/scripts/buildsystems/vcpkg.cmake",
        "VCPKG_TARGET_TRIPLET": "x64-windows"
    }
}

[Above] For Visual Studio Code, this is how to point a CMake project to the vcpkg CMake toolchain file. This file must be specified for any CMake project using vcpkg. Other IDEs or editors may have a different experience for pointing to CMake toolchain files.

Next, generate the CMake cache using the Visual Studio Code CMake: Configure command:

[cmakefileapi-driver] Removing d:/versions-test/build/CMakeCache.txt
[proc] Executing command: "C:\Program Files\CMake\bin\cmake.EXE" --no-warn-unused-cli -DCMAKE_TOOLCHAIN_FILE:STRING=D:/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET:STRING=x64-windows -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -Hd:/versions-test -Bd:/versions-test/build -G "Visual Studio 16 2019" -T host=x64 -A x64
[cmake] -- Running vcpkg install
[cmake] Detecting compiler hash for triplet x64-windows...
[cmake] The following packages will be built and installed:
[cmake]     fmt[core]:x64-windows -> 7.1.3 -- D:\vcpkg\buildtrees\versioning\versions\fmt\d99b6a35e1406ba6b6e09d719bebd086f83ed5f3
[cmake]     zlib[core]:x64-windows -> 1.2.11#9 -- D:\vcpkg\buildtrees\versioning\versions\zlib\827111046e37c98153d9d82bb6fa4183b6d728e4

[Above] This is the output produced by the CMake command line when generating the CMake cache. In Visual Studio Code, with the CMake Tools extension installed, the cache can be generated with the “CMake: Configure” command.

You will know that versioning is working if you notice the “<path-to-vcpkg>/buildtrees/versioning/versions/<portname>/<sha>” pattern in the paths. The port files are being checked out by vcpkg for the declared versions at those locations.

Lastly, run the program:

fmt version is 70103  
zlib version is 1.2.11

Example #2: Pinning older versions

Since baselines establish a version floor for all packages and explicit constraints get upgraded when they are lower than the baseline, we need another mechanism to downgrade versions past the baseline.

The mechanism vcpkg provides for that scenario is overrides. When an override is declared on a package, vcpkg will ignore all other version constraints either directly declared in the manifest or from transitive dependencies. In short, overrides will force vcpkg to use the exact version declared, period.

First, change your manifest to add an override on fmt and force vcpkg to use version 6.0.0:

{
    "name": "versions-test",
    "version": "1.0.0",
    "dependencies": [
        {
            "name": "fmt",
            "version>=": "7.1.3"
        },
        "zlib"
    ],
    "builtin-baseline": "b60f003ccf5fe8613d029f49f835c8929a66eb61", 
    "overrides": [
        { "name": "fmt", "version": "6.0.0" }
    ]
}

Next, delete your build folder, generate the CMake cache, and build again:

[cmake] -- Running vcpkg install
[cmake] Detecting compiler hash for triplet x64-windows...
[cmake] The following packages will be rebuilt:
[cmake]     fmt[core]:x64-windows -> 6.0.0 -- D:\Work\viromer\versioning\vcpkg\buildtrees\versioning\versions\fmt\d99b6a35e1406ba6b6e09d719bebd086f83ed5f3

Lastly, run the program:

fmt version is 60000  
zlib version is 1.2.11

How versioning works in vcpkg

In the diagram below we depict the chronology of versions released for four different libraries: FMT, zlib, Boost and Azure’s core C++ library.

Image vcpkg versioning diagram

The vcpkg versioning system is comprised of the following pieces:

Baselines

To enable versioning, it is required that you set a baseline to a specific vcpkg commit. By selecting a baseline, you are selecting a snapshot of a certain point in time of the chronology. Setting the baseline will set a minimum version floor on all your dependencies, the minimum versions being the ones that existed at the selected snapshot. In the picture above, we would have version 7.1.0 for FMT, version 1.2.11 at port revision 9 for zlib, version 1.74.0 for Boost and the September 2020 release for Azure’s Core C++ library. An advantage of using baselines is that versions that are contemporary are more likely to be compatible. The main vcpkg registry does, after all, build all libraries it contains at the same time to try to ensure compatibility of the whole catalog at every commit.

Constraints

In contrast with baselines that set minimum versions for all packages, constraints allow you to specify a minimum version in a package-by-package basis.

Constraints only allow you to upgrade versions further than those at the baseline. As stated above, the baseline sets a minimum version floor for all packages, so if you attempt to add a constraint that is lower than the baseline, the constraint will be upgraded.

An important thing to notice about constraints is that they are transitive where baselines and overrides are not. Dependencies in the graph can express their own constraints and they will be considered by the version resolution algorithm.

But what if you really need to set a version to be lower than the baseline? How can you do it without lowering the baseline? For those cases, you can use overrides.

Overrides

An override forces vcpkg to use a specific version while ignoring all other constraint (either explicit or transitive). This allows the user to solve some specific situations like:

  • Downgrading versions lower than the baseline.
  • Forcing upgraded/downgraded versions on transitive dependencies.
  • Solving version conflicts between different packages.

In the scenario depicted in the diagram, given all that we know about baselines, constraints, and overrides. We can see that version resolution for a project using all four packages would result in:

  • FMT: Version 7.1.2, as specified via constraint.
  • zlib: Version 1.2.11 at port revision 2, as specified via override.
  • Boost: Version 1.74, as defaulted by the baseline.
  • azure-core-pp: Version 2020-09-01, as defaulted by the baseline.

 

Versions and custom ports

The last thing to discuss is how overlay ports interact with versioning resolution. The answer is: they do not interact at all by design. Going into more detail, when you provide an overlay for a port, vcpkg will always use the overlay port without caring what version is contained in it. The reasons are two-fold: (1) it is consistent with the existing behavior of overlay ports (completely masking the existing port), and (2) overlay ports do not (and are not expected to) provide enough information to power vcpkg’s versioning feature.

If you want to have flexible port customization along with versioning features, you should consider making your own custom registry. See our registries specification for more details.

 

Further reading

If you’re interested in delving deeper into the details of how versioning works we recommended that you read the original versioning specification.

 

Give us your feedback!

Try out vcpkg by visiting our GitHub repo. We welcome your feedback on the tool and the new features in our issue tracker. To see what’s next for vcpkg, including support for versioning and registries, check out our roadmap.

Posted in C++

8 comments

Discussion is closed. Login to edit/delete existing comments.

  • Mats Taraldsvik 0

    The new features of vcpkg are great! Thank you for working on this!

    One thing confuses me a bit:
    > If you want to have flexible port customization along with versioning features, you should consider making your own custom registry. See our registries specification for more details.

    In the quote above, you link to the registries-2.md (second version) spec, while the roadmap links to the registries.md (first/original version). Is there an error in the roadmap link?

    It does say February 2021 in the roadmap for registries as well – does this mean that you intend to release registries this month as well? 🙂

  • Moritz Gerl 0

    Awesome, thank you!

  • Seyyed Hossein Hasanpour 0

    Thanks a lot, really appreciate your hard work and dedication.
    Would it not be better to remove the “builtin-baseline” completely and do a search internally yourself?
    This really shouldn’t be exposed to the user imho. How am I supposed to find which commit has some specific version of the library I’m looking for other than painstakingly browsing the commit logs?
    Also I believe specifying the exact version (version=) is also a necessity and it would make life much easier. I’m not sure why you left this out.
    You specify a version, you look into the commit in which that version was available, and then pull all the dependencies specified there.
    Its more straight forward and in line with other package managers specially the ones used in Python for example.

  • Scott Bailey 0
    {
        "name": "versions-test",
        "version": "1.0.0",
        "dependencies": [
            {
                "name": "fmt",
                "version>=": "7.1.3"     <----------------------- Why does this exist...
            },
            "zlib"
        ],
        "builtin-baseline": "b60f003ccf5fe8613d029f49f835c8929a66eb61", 
        "overrides": [
            { "name": "fmt", "version": "6.0.0" }   <---------------------- ...when an override supersedes it? This confuses me.
        ]
    }

    Why is ‘fmt’ set to ‘version >= 7.1.3’ and then overridden to ‘version = 6.0.0’? A good tool warns the user they’ve called out specific and conflicting version info in the manifest!!

  • ThunderSnow 0

    So versioning is great and all, but this makes it sound like vcpkg is moving more towards how Conan works with versioning. The main difference now being vcpkg is a centralized repository of code that creates packages and Conan is a de-centralized code base that creates packages. (This also goes against one of the major bullet points that is outlined in the vcpkg Docs as to why we should use vcpkg over Conan.)

    • Augustin PopaMicrosoft employee 0

      Hi, to be clear, versioning is an opt-in, optional feature. If you do not specify any versions all your dependencies are acquired and resolved automatically based on the current state of the catalog, with the usual guarantees for compatibility. The purpose of enabling versioning is to enable teams that have a hard requirement of locking themselves on specific versions, usually because they are on a larger team or work with a large codebase where they can’t be very flexible with that. We will continue to the default to the experience we’ve always had going forward, this is just a way to extend the vcpkg if you have stricter requirements for your dependency graph.

      • ThunderSnow 0

        Gotcha, thank you for the clarification!!

Feedback usabilla icon