February 15th, 2024

End of SSH-RSA support for Azure Repos

Bohdan Janousek
Bohdan Janousek
Senior Program Manager

Azure Repos provides two methods for users to access a git repository in Azure Repos – HTTPS and SSH. To use SSH, you need to create a key pair using one of the supported encryption methods. In the past we’ve been supporting only SSH-RSA and we’ve asked users to enable the SSH-RSA here. This is not required to be done anymore as in 2022 we’ve added support for the RSA-SHA2-256 and RSA-SHA2-512 to Azure DevOps Service. Later that year, the same support was also added to Azure DevOps Server 2022 and in August 2023 to Azure DevOps Server 2020 and 2019. The relevant release notes are linked here:

We are now announcing the deprecation of SSH-RSA as a supported encryption method for connecting to Azure Repos using SSH.

The SSH-RSA is a weak encryption method. It is also already deprecated by OpenSSH and cannot be used unless enabled explicitly.

This change impacts you immediately if you are using Azure DevOps Service and are using SSH-RSA keys to connect to repos through SSH.

If you use Azure DevOps Server, then this change does not currently impact you. However, this change will impact you when you upgrade to the next version of Azure DevOps Server 2022.3 (expected to be released towards the end of 2024), since that version will not support SSH-RSA keys. If you use any of the following versions of Azure DevOps Server, you are strongly encouraged to move from SSH-RSA keys to more secure RSA-SHA2-256 or RSA-SHA2-512 keys:

  • Azure DevOps Server 2019 Update 1.2 Patch 4 and later
  • Azure DevOps Server 2020 Update 1.2 Patch 7 and later
  • Azure DevOps Server 2022

Migrating to more secure ciphers as supported by current versions of Azure DevOps Server will prevent issues in the future when you upgrade to newer versions of the server.

The deprecation of SSH-RSA ciphers in Azure DevOps Service will be done in four phases as explained below.

Phase I – User opt-in

Any user of Azure DevOps Service can migrate from SSH-RSA to more secure ciphers supported by Azure Repos. These are RSA-SHA2-256 or RSA-SHA2-512. To do so, users can follow these steps:

  1. Generate new public private key either by running ssh-keygen -t rsa-sha2-256 or ssh-keygen -t rsa-sha2-512.
  2. Add the key generated in point 1 to the SSH agent. This can be done by running command ssh-add <PathToYourPrivateKey>.
  3. Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.
  4. Upload the public part of the key generated in point 1 to Azure DevOps. See this how to do so.

Phase II – Throttling/delaying

Starting early March 2024, we will start to delay any SSH operation where the SSH-RSA was used to secure the SSH channel. There will be a warning shown in the command line output stating:

“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase III – Brown out

In April 2024, we will start to fail the executions of any operation where the SSH-RSA was used to secure the channel. We will execute the failures in a couple of stages each with different number of failure intervals and different length of the individual interval. The intervals will start at random times during the day. Each stage will last for about a week.

Stage Interval length Count of intervals in a day Total failure time in a day
1 30 minutes 1 30 minutes
2 1 hour 3 3 hours
3 2 hours 4 8 hours
4 1 hour 12 12 hours

To ensure it is clear to user why the SSH operation failed we will add an error message to command line output. The error message will be:

“You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally. Any SSH session using SSH-RSA is subject to brown out (failure during random time periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

Phase IV – SSH-RSA removal

Late in Q2 2024, we will start failing the execution of any operation where the SSH-RSA was used to secure the SSH channel. To ensure it is clear to user why the SSH operation failed we will add to command line output the following error message:

“You’re using ssh-rsa that is unsupported. Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

FAQ

Q: I am running the Azure DevOps Server. Can I use SSH-RSA going forward?
A: Yes, you can for now. Nonetheless, when you are on any version of Azure DevOps Server supporting new ciphers, then you should consider moving to these more secure ones. See the Phase I – User opt-in chapter for more details how to do so.

Q: I am running Azure DevOps Server version without support for the new ciphers, but I do want to use a more secure cipher than SSH-RSA. Can I do that?
A: Yes, but you need to upgrade to any version of Azure DevOps Server supporting these more secure ciphers.

Q: Are there plans to remove support for the SSH-RSA also from the Azure DevOps Server?
A: Yes. In a future version we will remove the support for SSH-RSA from Azure DevOps Server. So, please consider moving to a more secure cipher. See the Phase I – User opt-in chapter for more details how to do so.

Q: Will I be able to use the SSH-RSA to connect to Azure DevOps Server after you remove support for SSH-RSA?
A: Yes, but this will require your organization administrators or IT specialists to take special action. Without that, you will be able to use only the supported ciphers.

Category
Azure & CloudDevOpsGit & Version Control

Author

Bohdan Janousek
Bohdan Janousek
Senior Program Manager

Bohdan has joined the Azure DevOps team in March 2022.

45 comments

Discussion is closed. Login to edit/delete existing comments.

Newest
Newest Popular Oldest
  • Dávid Rózsa · Edited

    In my opinion the brownout process causes unnecessary errors. During the brownout the server response still contains ssh-rsa:

    server proposal: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512

    This means that even if after the removal everything would work(as the response would only contain rsa-sha2-256,rsa-sha2-512) it will still result in an error during brownout unless ssh-rsa is explicitly disabled on client side, as a lot of clients will default to that if its available.

    It would make sense for me that during...

    Read more
    • Tomasz Bondarewicz · Edited

      I fully agree.
      I suppose they suppose, that if you support ssh-rsa then you don't support anything more secure and should be warned with denying your access to the repo. Which assumption is completely wrong.

      But, taking the reported (and experienced) brownouts randomness - how can you be sure, that cited server proposal have reached the "brownout'ed" endpoint? ;-)

      The implementation of this change is just messed and our complaints seem to be misunderstood (or ignored) :-(

      Read more
      • Eric Newton

        Another example of how botched this rollout is. I simply dont think they actually understand what they are even talking about, given the lack of information and downright incorrect information on how to generate a proper new key.

  • Snyder, George D.

    Honest Question: What’s the best way to hear about these sorts of things before they happen? To us, we didn’t know anything about it until our systems/processes were down due to not being able to reach ADO. This effectively got the message to us, but at the cost of down-time on our end.

    Should I be subscribing for notifications somewhere so that I can know about these brownouts?

    • Dahl, Tim

      It's 5:30am, May 1st, 2024 in Atlanta, GA USA. Moments after the ArgoCD patch was delivered yesterday the brownouts stopped. The prophecy of total SSH-RSA blackout on May 1st is, at the moment, not true.

      We applied the ArgoCD patch across all of our [Dev, QA, Prod] environments and then reverted Dev back. We wanted to see Dev fail and all other environments work today.

      Everything is working...

      Reminds me of 2012 when the Mayan...

      Read more
    • Eric Newton

      Good luck with that. Every one of Microsofts teams wants control over their own messaging so it seems like every single one does their own thing.

      The warning messaging shouldve been going out since at least Jan-2024, I only started seeing it in March 2024. Removing ciphers is a big deal. Problem i'm facing is the system Jenkins is running on cant be upgraded without a lot of pain. And we were...

      Read more
      • Snyder, George D.

        Thanks for sharing your experience with this. I have also been having trouble figuring out how to test that we've resolved it on our end and agree it is difficult without knowing when the brownouts will happen, or having a test endpoint to go after.

        But also, I don't know what I don't know. Maybe they do have a test endpoint that we could go after to confirm we have it working, and maybe...

        Read more
      • Tomasz Bondarewicz

        Maybe there's a way to open a support ticket in Azure DevOps itself? This should probably give you more control over the information exchange process and a better chance of getting your questions answered. At the same time, if individual user reports were handled, information about the problems, their scale and solutions would not be available globally, which I consider a negative aspect. It seems that our complaints in this "just a blog post" :-)...

        Read more
      • Eric Newton

        Why publish a blog at all then, if the author of the article doesnt monitor comments on his own blog post.

        Making core connection changes is a big deal, and this style of seemingly random brownouts does nothing to help diagnose and setup for the future.

        The proper way wouldve been to setup a new endpoint that 100% prevents the old rsa-ssh (SHA1). While the old endpoint would be generating warnings for any connections using rsa-ssh...

        Read more
  • Eric Newton

    Now again at 8:00 pm i’m not able to continue working for a f**king hour. THANKS GUYS.

  • Eric Newton · Edited

    Really having a devil of a time trying to get jenkins on ubuntu 14 to work. It doesnt work and then it works and then it doesnt work again.

    Thanks for the nondeterministic testing ability. Couldn't even give a number of seconds the window would be denying requests??? Had you reported the number of seconds the window would be denying requests, then AT LEAST i'd have something tangible to work with.

    I do things...

    Read more
  • Mcneal, Evan B

    For ArgoCD users that are experiencing pain during the recent random "brownouts", it seems that the ArgoCD Team is aware of an issue as outlined here: https://github.com/argoproj/argo-cd/issues/17634

    Using ecdsa or ED25519 would be ideal but Azure DevOps only supports rsa2-512 and 256, so we are waiting on either a fix from ArgoCD to explicitly support those or Microsoft to actually support new standard ssh tokens.

    @microsoft please consider supporting more than rsa in the future as this...

    Read more
  • Lucas Gonçalves

    Can you please prorogate a little bit these Brown out? We discover right now this “issue” and we need a bunch of ArgoCD to updates and there is a issue open for that:

    https://github.com/argoproj/argo-cd/issues/17634

  • Panos Sdonas

    Anyone knows when is the next brown out period? We can’t tell if our changes worked or not at the moment.

  • Tomasz Bondarewicz

    Hi,
    The way this change is being introduced unfortunately, in my opinion, complicates error analysis. Because when my software, which authenticates to the Azure DevOps repository, encounters errors, I don't know how to pinpoint the exact cause of the problem or how to test fixes.
    Errors occur randomly. Time windows can start and end at any moment, and during their duration, not all connection attempts result in an error.
    If it starts to work -...

    Read more
    • Bohdan JanousekMicrosoft employee Author

      ssh-keygen support (of course) to generate the desired signature type when signing certificates using an RSA CA key. The available RSA signature variants are “ssh-rsa” SHA1 signatures, not recommended), “rsa-sha2-256”, and “rsa-sha2-512” (the default). See https://www.man7.org/linux/man-pages/man1/ssh-keygen.1.html for more details.

      Step 3 of the opt-in phase explicitly asked to "Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used...

      Read more
      • Tomasz Bondarewicz

        What about environments where there is restricted (or none) access to local SSH configuration, like AKS worker nodes ?
        Someone can also assume, that the “local SSH configuration” regards the OS-level ssh config, while it can be as well the ssh implementation in installed software – like (as I suppose) ssh go library in FluxCD.

        Random, undocumented brown-out windows make it nightmare even more.

      • Rouke Broersma

        @bohdan The link you sent specifically says this about the -t option:

        <code>

        <code>

        This indicates to me that this is not the most relevant step in the guide. The more secure option is after all already the default so most people will have keys using the secure signature algorithm.

        To me this makes Step 3 the most important step in the guide, yet this is the step with the least information. It would be great if you could...

        Read more
      • Pablo Patruno

        Hi, did you get any luck ? I was struggling with this all the day . Even though I generate the key with the 256 or 512 signature I am still getting the same error .

    • Peter Ernst

      We have a similar thing happening with ArgoCD. Debugging this is extremely difficult since there is no way to check if our changes really fixed the issue, or if the Brown-out-Window has closed. Additionally we have been unable to restrict the used Algorithms through standard Linux methods inside of the container.
      I really like the idea of a endpoint, where the ssh-rsa HostKeyAlgorithm is disabled, or some other way such that you can reproduce this...

      Read more
      • Alex Movergan · Edited

        Same nightmare with FluxCD. I've regenerated keys like 100 times. Same story. The error won't go away.

        UPD: I tried to set the flux source controller to use , and now I can't checkout with a new error:

        <code>

        UPD2: fixed it by setting

        <code>

        Read more
      • Tomasz Bondarewicz

        Thanks @Alex Movergan !

        @Bohdan Janousek I still think that you are not doing this deprecation right :-/ Info about key regeneration is misleading in the first place. Second are the random brown-out windows.

Read next