{"id":27658,"date":"2021-02-25T21:49:24","date_gmt":"2021-02-25T21:49:24","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/cppblog\/?p=27658"},"modified":"2024-04-01T17:55:15","modified_gmt":"2024-04-01T17:55:15","slug":"registries-bring-your-own-libraries-to-vcpkg","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/cppblog\/registries-bring-your-own-libraries-to-vcpkg\/","title":{"rendered":"Registries: Bring your own libraries to vcpkg"},"content":{"rendered":"<p><em>Special thanks to Nicole Mazzuca for providing the content of this blog post. <\/em><\/p>\n<p>Are you working on a C++ project with library dependencies? Are you tired of maintaining a custom-built package management workflow with duct tape and git submodules? Then you should consider trying out a package manager. Perhaps you have been side-eyeing vcpkg for a while, and it looks like the perfect solution for your team, but for one problem: not all your dependencies are open source! Your company has internal libraries that they expect everyone to use too.<\/p>\n<p>&nbsp;<\/p>\n<h3>Can vcpkg work with non-open-source dependencies?<\/h3>\n<p>Yes! Up until now, your best options included hacking up overlay ports or forking the vcpkg ports tree. But there was room for improvement. Now, we are happy to announce a feature with an experience to manage any libraries you want, whether they are internal-only, open source, your own forks of open source projects, and more. In this blog post, we will delve into <strong>registries<\/strong>, our new experimental feature. We would love for you to try this feature out, give us feedback, and help us make it the best feature it can be!<\/p>\n<h3>Getting started with registries<\/h3>\n<p>So, we\u2019ve discussed the why of registries; now let\u2019s discuss the how. Let\u2019s say we are developers at North Wind Traders, and our company has a GitHub Enterprise subscription. Of course, depending on your company\u2019s or even your personal situation, you can use whatever solution you\u2019re already using. The goal in this blog post is to set up a git registry, the most common type of registry.<\/p>\n<h4>1. Create a new registry<\/h4>\n<p>The company\u2019s GitHub organization is at <a href=\"https:\/\/github.com\/northwindtraders\">https:\/\/github.com\/northwindtraders<\/a>, and that can be used to set up the registry. We will create our registry at <a href=\"https:\/\/github.com\/northwindtraders\/vcpkg-registry\">https:\/\/github.com\/northwindtraders\/vcpkg-registry<\/a>, since that\u2019s as good a name as any, and you can follow along with the branches there.<\/p>\n<p>Once we\u2019ve created this registry, we\u2019ll have to do a few things to actually set it up to contain the packages we want, in this case our internal JSON and Unicode libraries.<\/p>\n<ul>\n<li>First, we\u2019ll set up an empty baseline, the minimum requirement for a registry.<\/li>\n<li>Then, we\u2019ll add the files that our library needs to build, and make sure that they work.<\/li>\n<li>Finally, we\u2019ll add the library to the registry, by adding it to the versions database and writing down where to find the library in the git repository.<\/li>\n<\/ul>\n<h4>2. Create an empty registry baseline<\/h4>\n<p>So, let\u2019s start. Clone the (empty) repository, and add a baseline.json file underneath the top level versions directory, containing just the following:<\/p>\n<pre class=\"prettyprint\">{\r\n\"default\": {}\r\n}<\/pre>\n<h4>3. Create a vcpkg port for your library<\/h4>\n<p>Now, let\u2019s set up a port entry for the first of our two libraries, the Unicode library <a href=\"https:\/\/github.com\/northwindtraders\/beicode\">beicode<\/a>. If you have ever written a port, you know how to do this, but for those of us who haven\u2019t, let\u2019s go through it all the same.<\/p>\n<p>We first create a folder for the ports to live; following the standard of the vcpkg central registry, we\u2019ll call the folder ports. Since we use a stable git identifier to specify the directory, we don\u2019t need to put it in a specific place, but it\u2019s good to follow idioms. Inside this ports directory, create beicode\u2019s port directory; inside there, place two empty files, <em>portfile.cmake<\/em> and <em>vcpkg.json<\/em>.<\/p>\n<p>At this point, the registry directory should look something like this:<\/p>\n<pre class=\"prettyprint\">ports\/\r\n    beicode\/\r\n        portfile.cmake\r\n        vcpkg.json\r\nversions\/\r\n    baseline.json<\/pre>\n<p>Now, let\u2019s fill out the port. First, since the beicode GitHub repository already has a vcpkg.json manifest, copy that into the <em>vcpkg.json<\/em> file you created:<\/p>\n<pre class=\"prettyprint\">{\r\n  \"name\": \"beicode\",\r\n  \"version\": \"1.0.0\",\r\n  \"description\": \"A simple utf-8 based unicode decoding and encoding library\",\r\n  \"homepage\": \"https:\/\/github.com\/northwindtraders\/beicode\"\r\n}<\/pre>\n<h4>4. Test your new vcpkg port using overlays<\/h4>\n<p>Let\u2019s make sure this works by trying to install the port; we\u2019re not using registries yet, just the pre-existing overlay-ports feature to test stuff out:<\/p>\n<p><code>&gt; vcpkg install beicode --overlay-ports=vcpkg-registry\/ports\/beicode<\/code><\/p>\n<p>We should get an error: \u201c<em>The folder \/include is empty or not present<\/em>\u201d. Since we aren\u2019t doing anything just yet, that makes sense. So, let\u2019s fill out our port! Since our port is a simple CMake library, we can create a very simple <em>portfile.cmake<\/em>:<\/p>\n<pre class=\"prettyprint\">vcpkg_from_github(\r\n  OUT_SOURCE_PATH SOURCE_PATH\r\n  REPO northwindtraders\/beicode\r\n  REF 19a1f95c2f56a27ced90227b5e2754a602a08e69\r\n  SHA512 7b2bb7acb2a8ff07bff59cfa27247a7b2cced03828919cd65cc0c8cf1f724f5f1e947ed6992dcdbc913fb470694a52613d1861eaaadbf8903e94eb9cdfe4d000\r\n  HEAD_REF main\r\n)\r\n\r\nvcpkg_configure_cmake(\r\n  SOURCE_PATH \"${SOURCE_PATH}\"\r\n  PREFER_NINJA\r\n)\r\nvcpkg_install_cmake()\r\nvcpkg_fixup_cmake_targets()\r\n\r\nfile(REMOVE_RECURSE \"${CURRENT_PACKAGES_DIR}\/debug\/include\")\r\n\r\nfile(\r\n  INSTALL \"${SOURCE_PATH}\/LICENSE\"\r\n  DESTINATION \"${CURRENT_PACKAGES_DIR}\/share\/${PORT}\"\r\n  RENAME copyright)<\/pre>\n<p>If we run<\/p>\n<p><code>&gt; vcpkg install beicode --overlay-ports=vcpkg-registry\/ports\/beicode<\/code><\/p>\n<p>again, we\u2019ll see that it successfully installed! We have written our first port for the registry, and now all that there\u2019s left to do is to add the port to the version set in the registry.<\/p>\n<h4>5. Specify each version of your library in the registry<\/h4>\n<p>Every port\u2019s version data lives in its own file: <em>versions\/[first character]-\/[portname].json<\/em>. For example, the version data for <em>fmt<\/em> would live in <em>versions\/f-\/fmt.json<\/em>; the version data for <em>zlib<\/em> would live in <em>versions\/z-\/zlib.json<\/em>. So, for <em>beicode<\/em>, create <em>versions\/b-\/beicode.json<\/em>:<\/p>\n<pre class=\"prettyprint\">{\r\n  \"versions\": [\r\n    {\r\n      \"version\": \"1.0.0\",\r\n      \"git-tree\": \"\"\r\n    }\r\n  ]\r\n}<\/pre>\n<p>And add the following to <em>versions\/baseline.json<\/em>:<\/p>\n<pre class=\"prettyprint\">{\r\n  \"default\": {\r\n    \"beicode\": { \"baseline\": \"1.0.0\", \"port-version\": 0 }\r\n  }\r\n}<\/pre>\n<p>Finally, let\u2019s figure out what to put in that &#8220;<em>git-tree<\/em>&#8221; field. Do a git commit of the beicode port tree (but do not push), to make sure git knows about it:<\/p>\n<pre class=\"prettyprint\">&gt; git add ports\/beicode\r\n&gt; git commit -m \"[beicode] new port\"<\/pre>\n<p>Then get the tree identifier for that directory:<\/p>\n<p><code>&gt; git rev-parse HEAD:ports\/beicode<\/code><\/p>\n<p>You should get something like <code>7fb5482270b093d40ab8ac31db89da4f880f01ba<\/code>; put that in for the &#8220;<em>git-tree<\/em>&#8221; in <em>beicode.json<\/em>, and commit the new files:<\/p>\n<pre class=\"prettyprint\">&gt; git add versions\r\n&gt; git commit --amend --no-edit<\/pre>\n<p>And we should be done! The reason we have to do this slightly complex dance is so that we can grab exactly the files of the version we want; other versions will exist in the history of the repository, and thus are always there to be checked out.<\/p>\n<h4>6. Consume libraries from your vcpkg registry in a C++ project<\/h4>\n<p>Once we\u2019ve done this, let\u2019s try to consume the library from the new registry in an example codebase. Create a directory outside of the registry and change into that directory. Create a <em>vcpkg.json<\/em> which depends on beicode:<\/p>\n<pre class=\"prettyprint\">{\r\n\"name\": \"test\",\r\n\"version\": \"0\",\r\n\"dependencies\": [\r\n  \"fmt\",\r\n  \"beicode\"\r\n]\r\n}<\/pre>\n<p>And a <em>vcpkg-configuration.json<\/em> that sets up the registry as a git registry:<\/p>\n<pre class=\"prettyprint\">{\r\n  \"registries\": [\r\n    {\r\n      \"kind\": \"git\",\r\n      \"repository\": \"[full path to]\/vcpkg-registry\",\r\n      \"packages\": [ \"beicode\", \"beison\" ]\r\n    }\r\n  ]\r\n}<\/pre>\n<p>And try a vcpkg install:<\/p>\n<p><code>&gt; vcpkg install --feature-flags=registries,manifests<\/code><\/p>\n<p>If it works, then you\u2019re ready to push the registry upstream! You can try again with the actual remote registry by replacing the &#8220;<em>repository<\/em>&#8221; field in your <em>vcpkg-configuration.json<\/em> file with the actual upstream repository URL.<\/p>\n<p>&nbsp;<\/p>\n<h3>How vcpkg resolves libraries from registries<\/h3>\n<p>You\u2019ll notice that beicode and beison are taken from the registry we created; this is because we\u2019ve explicitly said in vcpkg-configuration.json that this is where they\u2019re from. Since we haven\u2019t said where fmt should come from, it just comes from the default registry, which is in this case the registry that ships with vcpkg itself. Registries are never transitive; if you left off beicode from the registry in vcpkg-configuration.json, this would fail to work since beicode doesn\u2019t exist in the default registry, and that\u2019s where vcpkg will look for it. If you wanted to override fmt with your own copy, you could add it to the registry, and then add it to the packages field.<\/p>\n<p>Packaging beison will be much the same, just with a different name. You can try it out for yourself, and then see if your code is any different from <a href=\"https:\/\/github.com\/northwindtraders\/vcpkg-registry\">the upstream<\/a>.<\/p>\n<p>&nbsp;<\/p>\n<h3>Give us your feedback!<\/h3>\n<p>Try out vcpkg by visiting our <a href=\"https:\/\/github.com\/microsoft\/vcpkg\/issues\">GitHub repo<\/a>. We welcome your feedback on the tool and the new features <a href=\"https:\/\/github.com\/microsoft\/vcpkg\/issues\">in our issue tracker<\/a> or by reaching out to us at <a href=\"mailto:vcpkg@microsoft.com\">vcpkg@microsoft.com<\/a>. We are always looking to improve your experience. To see what\u2019s next for vcpkg, <a href=\"https:\/\/aka.ms\/vcpkg\/roadmap\">check out our roadmap<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Special thanks to Nicole Mazzuca for providing the content of this blog post. Are you working on a C++ project with library dependencies? Are you tired of maintaining a custom-built package management workflow with duct tape and git submodules? Then you should consider trying out a package manager. Perhaps you have been side-eyeing vcpkg for [&hellip;]<\/p>\n","protected":false},"author":1063,"featured_media":35994,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,272],"tags":[273],"class_list":["post-27658","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cplusplus","category-vcpkg","tag-vcpkg"],"acf":[],"blog_post_summary":"<p>Special thanks to Nicole Mazzuca for providing the content of this blog post. Are you working on a C++ project with library dependencies? Are you tired of maintaining a custom-built package management workflow with duct tape and git submodules? Then you should consider trying out a package manager. Perhaps you have been side-eyeing vcpkg for [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27658","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/users\/1063"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/comments?post=27658"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/posts\/27658\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media\/35994"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/media?parent=27658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/categories?post=27658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/cppblog\/wp-json\/wp\/v2\/tags?post=27658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}