End of SSH-RSA support for Azure Repos
Azure Repos provides two methods for users to access a git repository in Azure Repos – HTTPS and SSH. To use SSH, you need to create a key pair using one of the supported encryption methods. In the past we’ve been supporting only SSH-RSA and we’ve asked users to enable the SSH-RSA here. This is not required to be done anymore as in 2022 we’ve added support for the RSA-SHA2-256 and RSA-SHA2-512 to Azure DevOps Service. Later that year, the same support was also added to Azure DevOps Server 2022 and in August 2023 to Azure DevOps Server 2020 and 2019. The relevant release notes are linked here:
- Azure DevOps Server 2022
- Azure DevOps Server 2020 Update 1.2 Patch 7
- Azure DevOps Server 2019 Update 1.2 Patch 4
We are now announcing the deprecation of SSH-RSA as a supported encryption method for connecting to Azure Repos using SSH.
The SSH-RSA is a weak encryption method. It is also already deprecated by OpenSSH and cannot be used unless enabled explicitly.
This change impacts you immediately if you are using Azure DevOps Service and are using SSH-RSA keys to connect to repos through SSH.
If you use Azure DevOps Server, then this change does not currently impact you. However, this change will impact you when you upgrade to the next version of Azure DevOps Server 2022.3 (expected to be released towards the end of 2024), since that version will not support SSH-RSA keys. If you use any of the following versions of Azure DevOps Server, you are strongly encouraged to move from SSH-RSA keys to more secure RSA-SHA2-256 or RSA-SHA2-512 keys:
- Azure DevOps Server 2019 Update 1.2 Patch 4 and later
- Azure DevOps Server 2020 Update 1.2 Patch 7 and later
- Azure DevOps Server 2022
Migrating to more secure ciphers as supported by current versions of Azure DevOps Server will prevent issues in the future when you upgrade to newer versions of the server.
The deprecation of SSH-RSA ciphers in Azure DevOps Service will be done in four phases as explained below.
Phase I – User opt-in
Any user of Azure DevOps Service can migrate from SSH-RSA to more secure ciphers supported by Azure Repos. These are RSA-SHA2-256 or RSA-SHA2-512. To do so, users can follow these steps:
- Generate new public private key either by running
ssh-keygen -t rsa-sha2-256orssh-keygen -t rsa-sha2-512. - Add the key generated in point 1 to the SSH agent. This can be done by running command
ssh-add <PathToYourPrivateKey>. - Change the local SSH configuration so the key generated in point 1 is listed before the SSH-RSA key. This is to ensure more secure algorithms will be used instead of SSH-RSA.
- Upload the public part of the key generated in point 1 to Azure DevOps. See this how to do so.
Phase II – Throttling/delaying
Starting early March 2024, we will start to delay any SSH operation where the SSH-RSA was used to secure the SSH channel. There will be a warning shown in the command line output stating:
“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”
Phase III – Brown out
In April 2024, we will start to fail the executions of any operation where the SSH-RSA was used to secure the channel. We will execute the failures in a couple of stages each with different number of failure intervals and different length of the individual interval. The intervals will start at random times during the day. Each stage will last for about a week.
| Stage | Interval length | Count of intervals in a day | Total failure time in a day |
|---|---|---|---|
| 1 | 30 minutes | 1 | 30 minutes |
| 2 | 1 hour | 3 | 3 hours |
| 3 | 2 hours | 4 | 8 hours |
| 4 | 1 hour | 12 | 12 hours |
To ensure it is clear to user why the SSH operation failed we will add an error message to command line output. The error message will be:
“You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally. Any SSH session using SSH-RSA is subject to brown out (failure during random time periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”
Phase IV – SSH-RSA removal
Late in Q2 2024, we will start failing the execution of any operation where the SSH-RSA was used to secure the SSH channel. To ensure it is clear to user why the SSH operation failed we will add to command line output the following error message:
“You’re using ssh-rsa that is unsupported. Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”
FAQ
Q: I am running the Azure DevOps Server. Can I use SSH-RSA going forward?
A: Yes, you can for now. Nonetheless, when you are on any version of Azure DevOps Server supporting new ciphers, then you should consider moving to these more secure ones. See the Phase I – User opt-in chapter for more details how to do so.
Q: I am running Azure DevOps Server version without support for the new ciphers, but I do want to use a more secure cipher than SSH-RSA. Can I do that?
A: Yes, but you need to upgrade to any version of Azure DevOps Server supporting these more secure ciphers.
Q: Are there plans to remove support for the SSH-RSA also from the Azure DevOps Server?
A: Yes. In a future version we will remove the support for SSH-RSA from Azure DevOps Server. So, please consider moving to a more secure cipher. See the Phase I – User opt-in chapter for more details how to do so.
Q: Will I be able to use the SSH-RSA to connect to Azure DevOps Server after you remove support for SSH-RSA?
A: Yes, but this will require your organization administrators or IT specialists to take special action. Without that, you will be able to use only the supported ciphers.
Light
Dark
45 comments
Honest Question: What’s the best way to hear about these sorts of things before they happen? To us, we didn’t know anything about it until our systems/processes were down due to not being able to reach ADO. This effectively got the message to us, but at the cost of down-time on our end.
Should I be subscribing for notifications somewhere so that I can know about these brownouts?
Good luck with that. Every one of Microsofts teams wants control over their own messaging so it seems like every single one does their own thing.
The warning messaging shouldve been going out since at least Jan-2024, I only started seeing it in March 2024. Removing ciphers is a big deal. Problem i’m facing is the system Jenkins is running on cant be upgraded without a lot of pain. And we were tring to get off of Jenkins, now Devops forces us to invest time into a system we’re trying to deprecate ourselves.
The rollout shouldve been to create a new endpoint that only supported the new stuff. Send out warnings for the old endpoint about old ciphers getting decommissioned, and that gives people the ability to start testing using the new endpoints and finding out what will work.
Instead we get this garbage policy of “selective, random brownouts” that give you no information of when it ends, so you start testing something, the window closes and your “fix” starts working but oh wait, its not in the brownout window… Absolutely terrible way to manage a rollout.
Thanks for sharing your experience with this. I have also been having trouble figuring out how to test that we’ve resolved it on our end and agree it is difficult without knowing when the brownouts will happen, or having a test endpoint to go after.
But also, I don’t know what I don’t know. Maybe they do have a test endpoint that we could go after to confirm we have it working, and maybe there is some centralized channel for messaging we just don’t know about?
I figure I won’t know unless I ask since other times with other things I’ve assumed a service was just bad all around and then came to realize they had provided for me, but that I had not looked in the right places or done what they were assuming I would do (which of course is partially their fault, but I definitely feel like it gets harder with larger products/companies – for me the main takeaway was that they were actively trying and heard my feedback).
Keeping my fingers crossed 🤞
Maybe there’s a way to open a support ticket in Azure DevOps itself? This should probably give you more control over the information exchange process and a better chance of getting your questions answered. At the same time, if individual user reports were handled, information about the problems, their scale and solutions would not be available globally, which I consider a negative aspect. It seems that our complaints in this “just a blog post” 🙂 did not attract much attention from Microsoft.
Sure, I understand and share your need for some sort of official channel to inform users about such changes. I wish we could live to see such a channel (or learn about it’s existence 🙂 )
Why publish a blog at all then, if the author of the article doesnt monitor comments on his own blog post.
Making core connection changes is a big deal, and this style of seemingly random brownouts does nothing to help diagnose and setup for the future.
The proper way wouldve been to setup a new endpoint that 100% prevents the old rsa-ssh (SHA1). While the old endpoint would be generating warnings for any connections using rsa-ssh (SHA1) for the forseeable future, along with a HARD DATE (not “late Q2 2024”) as to when it enforces.
Absolute piss poor project management IMO.
It’s 5:30am, May 1st, 2024 in Atlanta, GA USA. Moments after the ArgoCD patch was delivered yesterday the brownouts stopped. The prophecy of total SSH-RSA blackout on May 1st is, at the moment, not true.
We applied the ArgoCD patch across all of our [Dev, QA, Prod] environments and then reverted Dev back. We wanted to see Dev fail and all other environments work today.
Everything is working…
Reminds me of 2012 when the Mayan Calendar ended/reset…
Some feared Armageddon… Some hoped for enlightenment… There was a movie…
Some were engaged with Microsoft SEV A 24×7 support where they tirelessly and pro-actively looked into the concern… Some also scheduled on-call staff to be ready in case a change was noticed or a call came in…
But it was just another day…
So far…
In my opinion the brownout process causes unnecessary errors. During the brownout the server response still contains ssh-rsa:
server proposal: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
This means that even if after the removal everything would work(as the response would only contain rsa-sha2-256,rsa-sha2-512) it will still result in an error during brownout unless ssh-rsa is explicitly disabled on client side, as a lot of clients will default to that if its available.
It would make sense for me that during the brownout ssh-rsa is not returned as a valid option by the azure server.
I fully agree.
I suppose they suppose, that if you support ssh-rsa then you don’t support anything more secure and should be warned with denying your access to the repo. Which assumption is completely wrong.
But, taking the reported (and experienced) brownouts randomness – how can you be sure, that cited server proposal have reached the “brownout’ed” endpoint? 😉
The implementation of this change is just messed and our complaints seem to be misunderstood (or ignored) 🙁
Another example of how botched this rollout is. I simply dont think they actually understand what they are even talking about, given the lack of information and downright incorrect information on how to generate a proper new key.