Update (October 2022): Deprecation of weak cryptographic protocols has completed.
Deprecation of TLS 1.0 and 1.1 has completed for the dev.azure.com and *.visualstudio.com domains – all HTTPS connections to Azure DevOps for these domains now use TLS 1.2.
Azure DevOps team needed to partially rollback the previous release of TLS 1.0/1.1 deprecation that was run on Jan 31st, 2022. This was due to unexpected issues caused by the change. Here’s a link to the previous blog post related to that release.
TLS 1.0/1.1 deprecation applies to all HTTPS connections to Azure DevOps Services including web API, and git connections to https://dev.azure.com/orgname and https://orgname.visualstudio.com. This does not apply and will not impact the Self-Hosted product: Azure DevOps Server.
Currently, connections to IPv6 endpoints of our services are already on enforced TLS 1.2 so the deprecation is targeting IPv4 endpoints only.
We anticipate minimal impacts to our customers as more than 99.5% of connections made to Azure DevOps Services already use TLS 1.2. Clients have TLS 1.2-compatibility issues because of obsolete OS version or if available updates are not applied (applies for all Windows, macOS and Linux) or legacy .NET Framework installation or OS configuration prohibiting certain TLS cipher suites.
Important Dates
To help our customers identify Azure DevOps-facing TLS 1.2-incompatible software in their systems, we will temporarily disable support for TLS 1.0/1.1 for 12 hours on:
-
March 22, 2022 09:00 to 21:00 UTC for https://orgname.visualstudio.com. TLS 1.0/1.1 connections to Azure DevOps Services using https://orgname.visualstudio.com will temporarily fail to connect. Connections using https://dev.azure.com on TLS 1.0/1.1 will not be impacted.
-
March 24, 2022 09:00 to 21:00 UTC for https://dev.azure.com/orgname. TLS 1.0/1.1 connections to Azure DevOps Services using https://dev.azure.com/orgname will temporarily fail to connect. Connections using https://orgname.visualstudio.com on TLS 1.0/1.1 will not be impacted.
We will be starting permanent rollout from March 31, 2022
How to enable TLS 1.2?
As a key area of focus we recommend Self-Hosted Agents and tooling that calls Azure DevOps Services APIs.
Check these red flags first when looking for TLS 1.2-incompatible flows: – Windows Server 2012 R2 or lower used – .NET Framework 4.7+ not installed on computer where PowerShell or .NET applications call Azure DevOps API
It is key that on the OS one of the four TLS 1.2 cipher suites below is enabled:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (*)
- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (*)
NOTE: (*) Cipher suites supported by Windows Server 2012 R2
Quick check of TLS 1.2 compatibility
Following checks are based on querying site status.dev.azure.com. For this domain, the TLS 1.0/1.1 protocols have already been deprecated. Note that checking accessibility of this domain by browser instead of PowerShell is not equivalent. Browsers often use crypto libraries (such as OpenSSL) and thus circumvent the classic HTTP/TLS stack that other software uses.
Manual PowerShell
Run this code in PowerShell console:
(Invoke-WebRequest -Uri status.dev.azure.com -UseBasicParsing).StatusDescription
You have the problem if the above fails with “The underlying connection was closed: An unexpected error occurred on a send.”
YAML
To check your pipeline agents: create testing pipeline OR add this step to existing pipelines:
steps:
- task: AzurePowerShell@5
inputs:
ScriptType: 'InlineScript'
Inline: '"TLS 1.2 readiness check:"; (Invoke-WebRequest -Uri status.dev.azure.com -UseBasicParsing).StatusDescription'
FailOnStandardError: true
Pipeline task
This shows a release pipeline task that can be added to perform a check for TLS 1.2 readiness on your Self-Hosted Agents. If the check fails, the task will throw an error that can be actioned.
Troubleshooting
For Windows OS, TLS 1.2 is natively supported by all versions from Windows 7 / Windows Server 2008 SP2. However, even at TLS 1.2-compatible OS, issues may be caused by misconfigurations such as when all cipher suites accepted by Azure DevOps are disabled. This may be set up locally or via domain Group Policies.
At application level, capabilities provided by the OS may be narrowed down making the app to use less secure TLS setting. For example, .NET apps which are built against legacy .NET Framework versions may require certain measures to be taken to ensure that TLS 1.2 is favoured.
Based on learnings from supporting our customers, we’ve put together a troubleshooting script which performs TLS 1.2 probe, identifies most common misconfigurations and provides mitigation advice.
You can find the script on GitHub:
https://github.com/microsoft/azure-devops-tls12
The project is work in progress; feel free to contribute with PR or open an issue.
Conclusion
We apologize for any disruption this may cause and appreciate your support to improve our security posture. As always, if you have any questions or concerns related to this announcement, please do not hesitate to reach out to us on Developer Community or by posting your comments below.
good to know…
Was a final update to dev.azure.com done in the first week of May? At that time we noticed a marketplace extension that stopped functioning properly. The developer has since fixed the issue, but we need to go back and review data from over 30 projects to fix the data. Having a roll out date from that week, if one is available, would help greatly.
Hello Sarah, the global rollout of the final update to dev.azure.com took place at about 2022-05-03 14:36 UTC. Hope it helps!
It looks like https://orgname.visualstudio.com is still accepting TLS 1.0 & 1.1.
Is there a date when rollout will be completed?
Many of my users still use old URL form interchangeably or even preferentially … even after org settings long updated to use new URL.
Matt, this is correct – at the moment, IPv4 endpoints of visualstudio.com are the last ones remaining.
We roll this process slowly. After each step we’re closely monitoring health of the customers with higher legacy TLS exposure. Once visualstudio.com is switched, we will update here.
Thank you, it does!!
One of our build agents is no longer connecting. I tried to run the troubleshooting tool and Invoke-WebRequest fails but everything else is green. I ran Invoke-WebRequest against github and got a 200. Any advice?
Azure DevOps TLS 1.2 transition readiness checker v. 2022-04-14
| Probing Azure DevOps sites |
Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to 13.107.6.183:443. Error: Unable to read data from the transport connection: An exis
ting...
Hi Mustafa, good to see you checked the main things.
Seeing GitHub check works well, this leads to cipher suites problem. But before we go that route, let's rule-out .NET Framework / PS 5 having something to do with it. Can you please run the troubleshooting checker script on PowerShell 7? This will circumvent .NET Framework and run, same as Pipeline agent, on top of .NET Core.
If the PS 7 result is the same as with...
identical results in PS7
in that location we have a subfolder called Diffie-Hellman with the keys Default:Not Set and Enabled:0
thanks.
There seems to be some problem that prevents usage of ECDHE cipher suites despite they are enabled at usual places. (This can be confirmed by capturing TLS handshake using Wireshark and looking at ClientHello packet to verify if cipher suites , are missing in client's list.)
As a workaround to ECDHE problem you can try to enable DHE cipher suites which you have configured as disabled ("Diffie-Helman key exchange disabled."). Set to 0xFFFFFFFF (),...
Thanks Mustafa. The elliptic curves NistP256 and NistP384 are ones used by AzDO, so this looks good. I admit having no further ideas why ECDHE does not work in your setup. I advise you to reach out to Azure support to get more personalized help.
Regarding Diffie-Hellman disablement: if this is security motivated, then the recommended practice known to me is not to disable the Diffie-Hellman as such, but to enforce minimum server key length to...
Hello Ondrej,
See below for results
PS C:\Windows\System32> Get-TlsEccCurve
curve25519
NistP256
NistP384
Hello Ondrej,
Trying to fix the DHE caused the server to go into an unstable state (guessing the group policy didnt like what I did). I’m waiting for the server to be restored so I can try next steps.
Thanks for all your help so far, I’ll reply back once I’m able to run Get-TlsEccCurve
The ECDHE problem may be caused by missing elliptic curves. Can you post here output of
Get-TlsEccCurve?Mark, I think your "TLS powershell check" should also include -UseBasicParsing. Without it, I got the following error:
<code>
Which I was unable to resolve. But thanks to https://stackoverflow.com/a/38054505/2246411 I found that the following is a better option:
<code>
I also found that the following pipeline task was much shorter (and I believe it's equivalent):
<code>
Thanks, Derek for the comments on the Invote-WebRequest statement. I’ll add it to the blog post, as anything to help save people time when looking at this is great. Much appreciated.
Is there a target date for when the rollout be completed? I.E when will TLS 1.1 & 1.0 no longer be supported
dev.azure.com has been completed now. If you’re using this domain, then you should be on TLS 1.2. visualstudio.com is currently being worked on.
Our Windows 2016 private agent pool stopped working a while back and we currently have a support case open to troubleshoot. The image is based on the scripts made available through Microsofts Virtual Environments public GitHub repo.
We have a hub-spoke network based on vWan with an AzFW for outbound traffic (connected vNet). In the same subnet we have the w2016 scale set, a VM based on the same w2016 image and a w2022 VM. Both...
Answering these questions may help you in investigation:
- When you run the script, does it say "Failed to reach the destination. This is connectivity or DNS problem, not TLS compatibility issue." or does it say "ISSUE FOUND: This may be TLS compatibility issue!"
- regarding the network trace. Since you say Client Hello did not receive response, it sounds that TCP connecition establishment preceeding the TLS handshake was successfull (i.e. SYN and...
Hello Ondrej,
Thank you for your response, much appreciated. I added the answers to your questions inline:
– When you run the script, does it say “Failed to reach the destination. This is connectivity or DNS problem, not TLS compatibility issue.” or does it say “ISSUE FOUND: This may be TLS compatibility issue!”
-> Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to 13.107.6.183:443. Error: Unable to read data from the...
Hi D.K. sorry to see you still face the issue.
The packet sequence definitelly indicates middleman (read: firewall) cutting the connection based on what it sees in ClientHello. I would focus investigation this way.
The paradox you see with WS 2022 is a good clue. What WS 2022 may do differently than your WS 2k12 VMs:
- it may favor IPv6 when resolving the DNS names and thus other set of FW rules may kick-in.
- WS...
Thank you Ondrej, we’ll investigate further 🙂
Can you tell us if this activity has started. Last time on 24th March we were not able to connect. Now we have made some cipher related changes. We are running windows 2012 R2. We want to check if our changes is working.
So we were wondering if you can tell us when this blocking will begin again. While we are trying to migrate to new servers, it will take some time.
Hi Anup. The start of the deprecation process was set to yesterday but we're not switching all our endpoints at once. Take it that TLS 1.0/1.1 support guarantee ended with the end of March. The process is phased to take several weeks, we'll update here when important milestones are achieved.
If you need to check anytime what is the momentarily state of TLS setup for the Azure DevOps Services domain of your choice, you can use...
Hi Mark, We have been working an MS Premier support case since last Thursday and only today have a POSSIBLE fix for our environment. Is there any way we can get a 30 day extension so that we can properly test and apply our fix?
We also had a Github Issue going at the same time and I'll be posting our fix details there.
Essentially, for us it came down to a Domain level group policy that...
Hi Andrew
The process of deprecation has started, but it is not atomic and will take time. We're happy to hear that together with Premier support you identified the fix. We are interested in further challenges you face and we're to continue helping you via Premium support. We will pass this feedback to them.
Ondrej has added more detail in a reply below: link
Why is this next one a red flag?
“.NET Framework 4.7+ not installed on computer where PowerShell or .NET applications call Azure DevOps API”
because NET Framework 4.6.2 and later supports TLS 1.1 and TLS 1.2.
Source: https://docs.microsoft.com/en-us/mem/configmgr/core/plan-design/security/enable-tls-1-2-client#install-net-updates
What am I missing here?
Arien, thank you, I understand confusion in this territory.
We followed advisory from this article by .NET team. There is mentioned a case where upgrading to 4.7+ is suggested for apps targeting specific legacy versions of .NET Framework to work properly with TLS 1.2.
getting this now
Windows PowerShell
Copyright (C) 2012 Microsoft Corporation. All rights reserved.
PS C:\Users\BizTalkDevelopersbx> cd B:\azure-devops-tls12-main
PS B:\azure-devops-tls12-main> ./AzureDevOpsTls12Analysis.ps1
| Probing Azure DevOps sites |
Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to . Error: Unable to read data from the transport connection: An exis
ting connection was forcibly closed by the remote host.
********************************************************************************
| Analysis of TLS 1.2 compatibility: OS |
PS Version: 3.0
PS Edition:
Win Build Version: 6.1.7601.65536
CLR Version:...
I ran the latest version of the script. All seems to check out, but still won't connect.
Azure DevOps TLS 1.2 transition readiness checker v. 2022-03-25
| Probing Azure DevOps sites |
Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to 13.107.6.183:443. Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
********************************************************************************
| Analysis of TLS 1.2 compatibility: OS |
PS Version: 5.1.14409.1018
PS Edition:...
Hi Geoff, I see WinServer 2012 R2 or Windows 8.1 is used
Are you getting the consistent results when running via PowerShell 7? (This way we circumvent .NET Framework since PS 7 is on Core). Please run both the readiness checker script and the simple script from the blog above ()
You also could look at this article https://docs.microsoft.com/en-us/windows/win32/secauthn/tls-cipher-suites-in-windows-8-1 (applies to both W8.1 and WS2012R2). On top of other advice in the article, there is a link...