{"id":15424,"date":"2024-05-10T00:00:00","date_gmt":"2024-05-10T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ise\/?p=15424"},"modified":"2024-07-18T11:58:25","modified_gmt":"2024-07-18T18:58:25","slug":"synchronizing-multiple-remote-git-repositories","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/synchronizing-multiple-remote-git-repositories\/","title":{"rendered":"Synchronizing multiple remote Git Repositories"},"content":{"rendered":"

In a recent project, we needed to synchronize multiple remote Git repositories. We were maintaining an open-source project on GitHub<\/a>, and a private downstream mirror on Azure Repos<\/a>. It was important for the project to have a safe and reliable process that would allow us to keep the two repositories in sync.<\/p>\n

In our specific case, this would be a one-way synchronization from GitHub to Azure Repos. It was not possible to synchronize in the other direction, as the Azure Repos repository would contain organization-specific implementations that could not be shared publicly.<\/p>\n

We made the conscious decision to keep the synchronization manual. The project maintainers wanted control over the process. They wanted to review, and approve, any changes that were being moved downstream. This would also give the team an opportunity to resolve any conflicts that may arise.<\/p>\n

During the project, we discovered different ways to approach this problem. In the end we settled on a process that was simple, and easy to follow – with minimal knowledge of Git required. This post will outline the process we followed, and the steps we took to synchronize changes between remote repositories.<\/p>\n

Repository Setup<\/h2>\n

Before starting the synchronization process, we needed to set up the mirror on Azure Repos. This was a one-time operation, and was done using the Azure Repos web interface. This is a simple process, nicely documented here<\/a>.<\/p>\n

\"Importing<\/p>\n

[Source: Microsoft Learn<\/a>]<\/em><\/p>\n

Once the repository was imported, we could setup our local repository to have two remotes<\/em>. origin<\/code> pointing to Azure Repos, and upstream<\/code> pointing to GitHub.<\/p>\n

$ git clone <azure-devops-url> my-repo\r\n$ cd my-repo\r\n$ git remote -v\r\norigin      <azure-devops-url> (fetch)\r\norigin      <azure-devops-url> (push)\r\n$ git remote add upstream <github-url><\/code><\/pre>\n
\"Local<\/p>\n
At this point, our local repository was set up to push and pull changes from both origin<\/code> and upstream<\/code>. We could confirm this by running git remote -v<\/code>:<\/p>\n
$ git remote -v\r\norigin      <azure-devops-url> (fetch)\r\norigin      <azure-devops-url> (push)\r\nupstream    <github-url> (fetch)\r\nupstream    <github-url> (push)<\/code><\/pre>\n
(Optional)<\/strong> Manually import a repo using git CLI<\/h3>\n
We were fortunate to be able to use the web interface to import the initial repository. However, if you need to import a repository using the git<\/code> CLI, you can follow these steps:<\/p>\n
    \n
  1. \n
    Clone the source repo to a temporary folder on your computer using the --bare<\/code> option:<\/p>\n
    git clone --bare <github-url> bare-repo\r\ncd bare-repo<\/code><\/pre>\n
    \n
    Note:<\/strong> The --bare<\/code> option creates a bare repository, which does not have a working directory. This is useful when you want to import a repository into another repository.<\/p>\n<\/blockquote>\n<\/li>\n
  2. \n
    Create a new empty repository on Azure Repos.<\/p>\n<\/li>\n
  3. \n
    Push the source repository to the new Azure Repos repository:<\/p>\n
    git push --mirror <azure-devops-url><\/code><\/pre>\n
    \n
    Note:<\/strong> The --mirror<\/code> option pushes all branches, tags, and commits from the source repository to the target repository.<\/p>\n<\/blockquote>\n<\/li>\n<\/ol>\n
    Synchronizing Changes<\/h2>\n
    The synchronization process itself was straightforward. We would fetch changes from GitHub, create a new local branch, and push it to Azure Repos. This would allow us to create a Pull Request to review the changes, and merge them into the main<\/code> branch.<\/p>\n
    \"Sync<\/p>\n
    Let’s assume we had some updates<\/em> from upstream<\/code> we wanted to synchronize. This is the process we would follow:<\/p>\n
      \n
    1. \n
      Fetch changes from GitHub (upstream<\/code>):<\/p>\n
      git fetch upstream<\/code><\/pre>\n<\/li>\n
    2. \n
      Checkout a new local branch, based on GitHub’s main<\/code> branch:<\/p>\n
      git checkout --no-track -b chore\/upstream-sync upstream\/main<\/code><\/pre>\n
      \n
      Note:<\/strong> The --no-track<\/code> flag is used to create a new branch that is not tracking any remote branch.<\/p>\n<\/blockquote>\n<\/li>\n
    3. \n
      Merge the latest changes from Azure Repos (origin<\/code>) into chore\/upstream-sync<\/code>:<\/p>\n
      git fetch origin\r\ngit merge origin\/main\r\n# resolve conflicts, if any<\/code><\/pre>\n
      \n
      Note:<\/strong> At this stage, if you receive an error message “fatal: refusing to merge unrelated histories”, it’s likely that origin<\/code> did not originate from upstream<\/code>. Ensure that the repository was imported<\/em> correctly.<\/p>\n<\/blockquote>\n<\/li>\n
    4. \n
      Publish the new local branch to Azure Repos:<\/p>\n
      git push -u origin chore\/upstream-sync<\/code><\/pre>\n<\/li>\n
    5. \n
      Create a pull request<\/a> in Azure Repos to merge the chore\/upstream-sync<\/code> branch into the main<\/code> branch.<\/p>\n
      \"New<\/p>\n<\/li>\n
    6. \n
      After the PR meets all branch policies and has all required approvals, you can complete the PR.<\/p>\n
      \"Complete<\/p>\n<\/li>\n<\/ol>\n
      “Patch-test” from upstream<\/code><\/h2>\n
      During the project, we discovered a scenario where we needed to test changes in the downstream environment before merging them into GitHub’s main<\/code> branch. To achieve this, we needed to patch<\/em> the changes from GitHub into a temporary branch on Azure Repos. This would allow us to run organization-specific tests, and deployment pipelines.<\/p>\n
      \"Patch-test<\/p>\n
      This wasn’t a regular practice, but frequent enough to warrant a process:<\/p>\n
        \n
      1. \n
        Let’s assume we already have a feature branch, called feat\/my-new-feature<\/code> on GitHub.<\/p>\n<\/li>\n
      2. \n
        Create a new local patch-test\/<\/code> branch, based on the feature branch:<\/p>\n
        git checkout --no-track -b patch-test\/my-new-feature upstream\/feat\/my-new-feature<\/code><\/pre>\n<\/li>\n
      3. \n
        Merge the latest changes from Azure Repos (origin<\/code>) into patch-test\/my-new-feature<\/code>:<\/p>\n
        git fetch origin\r\ngit merge origin\/main<\/code><\/pre>\n
        \n
        At this point, it’s important to re-iterate that our synchronization process was one-way. Once a branch was “contaminated” with organization-specific changes, it could not be merged, or pushed, back into GitHub.<\/p>\n<\/blockquote>\n<\/li>\n
      4. \n
        Push patch-test\/my-new-feature<\/code> to Azure Repos (origin<\/code>).<\/p>\n
        git push -u origin patch-test\/my-new-feature<\/code><\/pre>\n<\/li>\n
      5. \n
        Run any necessary test and deployment pipelines on Azure DevOps from the patch-test\/my-new-feature<\/code> branch.<\/p>\n<\/li>\n<\/ol>\n
        But wait! What if I need to make a change?!<\/strong><\/p>\n
          \n
        1. \n
          It is important to keep new commits<\/em> isolated from the “contaminated” patch-test\/<\/code> branch. We start by switching to the original feature branch, that is tracked to upstream<\/code>:<\/p>\n
          git checkout feat\/my-new-feature<\/code><\/pre>\n<\/li>\n
        2. \n
          Commit whatever change you need.<\/p>\n<\/li>\n
        3. \n
          Then, repeat the merge<\/em> process:<\/p>\n
          git checkout patch-test\/my-new-feature\r\ngit merge feat\/my-new-feature\r\ngit push<\/code><\/pre>\n<\/li>\n
        4. \n
          Once you are happy with the changes, you can push the feature branch to GitHub, and delete the patch-test\/<\/code> branch.<\/p>\n
          git checkout feat\/my-new-feature\r\ngit push\r\ngit branch -d patch-test\/my-new-feature<\/code><\/pre>\n<\/li>\n<\/ol>\n
          Conclusion<\/h2>\n
          Synchronizing multiple remote Git repositories can be a daunting task. It certainly was for our team. However, the right process can provide a safe and reliable way to keep repositories in sync. With a few simple steps, maintainers and other users of the project, can confidently move open-source contributions downstream. Allowing them to maintain and update a private organization-specific implementation.<\/p>\n
          Note: The feature image used in this blog was created using AI in Microsoft Designer<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
          summary of the post<\/p>\n","protected":false},"author":158471,"featured_media":15427,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1,3451],"tags":[3520,3317,187],"class_list":["post-15424","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cse","category-ise","tag-azure-repos","tag-git","tag-github"],"acf":[],"blog_post_summary":"
          summary of the post<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/15424","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/158471"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=15424"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/15424\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/15427"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=15424"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=15424"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=15424"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}