Deprecating weak cryptographic standards (TLS 1.0 and 1.1) in Azure DevOps Services
Update (October 2022): Deprecation of weak cryptographic protocols has completed.
Deprecation of TLS 1.0 and 1.1 has completed for the dev.azure.com and *.visualstudio.com domains – all HTTPS connections to Azure DevOps for these domains now use TLS 1.2.
Azure DevOps team needed to partially rollback the previous release of TLS 1.0/1.1 deprecation that was run on Jan 31st, 2022. This was due to unexpected issues caused by the change. Here’s a link to the previous blog post related to that release.
TLS 1.0/1.1 deprecation applies to all HTTPS connections to Azure DevOps Services including web API, and git connections to https://dev.azure.com/orgname and https://orgname.visualstudio.com. This does not apply and will not impact the Self-Hosted product: Azure DevOps Server.
Currently, connections to IPv6 endpoints of our services are already on enforced TLS 1.2 so the deprecation is targeting IPv4 endpoints only.
We anticipate minimal impacts to our customers as more than 99.5% of connections made to Azure DevOps Services already use TLS 1.2. Clients have TLS 1.2-compatibility issues because of obsolete OS version or if available updates are not applied (applies for all Windows, macOS and Linux) or legacy .NET Framework installation or OS configuration prohibiting certain TLS cipher suites.
Important Dates
To help our customers identify Azure DevOps-facing TLS 1.2-incompatible software in their systems, we will temporarily disable support for TLS 1.0/1.1 for 12 hours on:
-
March 22, 2022 09:00 to 21:00 UTC for https://orgname.visualstudio.com. TLS 1.0/1.1 connections to Azure DevOps Services using https://orgname.visualstudio.com will temporarily fail to connect. Connections using https://dev.azure.com on TLS 1.0/1.1 will not be impacted.
-
March 24, 2022 09:00 to 21:00 UTC for https://dev.azure.com/orgname. TLS 1.0/1.1 connections to Azure DevOps Services using https://dev.azure.com/orgname will temporarily fail to connect. Connections using https://orgname.visualstudio.com on TLS 1.0/1.1 will not be impacted.
We will be starting permanent rollout from March 31, 2022
How to enable TLS 1.2?
As a key area of focus we recommend Self-Hosted Agents and tooling that calls Azure DevOps Services APIs.
Check these red flags first when looking for TLS 1.2-incompatible flows: – Windows Server 2012 R2 or lower used – .NET Framework 4.7+ not installed on computer where PowerShell or .NET applications call Azure DevOps API
It is key that on the OS one of the four TLS 1.2 cipher suites below is enabled:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (*)
- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (*)
NOTE: (*) Cipher suites supported by Windows Server 2012 R2
Quick check of TLS 1.2 compatibility
Following checks are based on querying site status.dev.azure.com. For this domain, the TLS 1.0/1.1 protocols have already been deprecated. Note that checking accessibility of this domain by browser instead of PowerShell is not equivalent. Browsers often use crypto libraries (such as OpenSSL) and thus circumvent the classic HTTP/TLS stack that other software uses.
Manual PowerShell
Run this code in PowerShell console:
(Invoke-WebRequest -Uri status.dev.azure.com -UseBasicParsing).StatusDescription
You have the problem if the above fails with “The underlying connection was closed: An unexpected error occurred on a send.”
YAML
To check your pipeline agents: create testing pipeline OR add this step to existing pipelines:
steps:
- task: AzurePowerShell@5
inputs:
ScriptType: 'InlineScript'
Inline: '"TLS 1.2 readiness check:"; (Invoke-WebRequest -Uri status.dev.azure.com -UseBasicParsing).StatusDescription'
FailOnStandardError: true
Pipeline task
This shows a release pipeline task that can be added to perform a check for TLS 1.2 readiness on your Self-Hosted Agents. If the check fails, the task will throw an error that can be actioned.
Troubleshooting
For Windows OS, TLS 1.2 is natively supported by all versions from Windows 7 / Windows Server 2008 SP2. However, even at TLS 1.2-compatible OS, issues may be caused by misconfigurations such as when all cipher suites accepted by Azure DevOps are disabled. This may be set up locally or via domain Group Policies.
At application level, capabilities provided by the OS may be narrowed down making the app to use less secure TLS setting. For example, .NET apps which are built against legacy .NET Framework versions may require certain measures to be taken to ensure that TLS 1.2 is favoured.
Based on learnings from supporting our customers, we’ve put together a troubleshooting script which performs TLS 1.2 probe, identifies most common misconfigurations and provides mitigation advice.
You can find the script on GitHub:
https://github.com/microsoft/azure-devops-tls12
The project is work in progress; feel free to contribute with PR or open an issue.
Conclusion
We apologize for any disruption this may cause and appreciate your support to improve our security posture. As always, if you have any questions or concerns related to this announcement, please do not hesitate to reach out to us on Developer Community or by posting your comments below.
Light
Dark
87 comments
Our Windows 2016 private agent pool stopped working a while back and we currently have a support case open to troubleshoot. The image is based on the scripts made available through Microsofts Virtual Environments public GitHub repo.
We have a hub-spoke network based on vWan with an AzFW for outbound traffic (connected vNet). In the same subnet we have the w2016 scale set, a VM based on the same w2016 image and a w2022 VM. Both w2016 hosts fail the PowerShell script (TLS 1.2 client usage enabled, cipher suites required by Azure DevOps enabled on machine). but the w2022 VM passes the test.
We also have the same w2016 image deployed in an isolated vNet (not connected, default NSG, no UDR). On this VM the powershell script passes the test.
A network trace shows that the w2016 VMs in the connected vNet receive no response to the TLS 1.2 Client Hello message, though the w2022 VM does.
Is this a network issue or a configuration issue on the VMs? Both scenarios have contradictions.
Answering these questions may help you in investigation:
– When you run the script, does it say “Failed to reach the destination. This is connectivity or DNS problem, not TLS compatibility issue.” or does it say “ISSUE FOUND: This may be TLS compatibility issue!”
– regarding the network trace. Since you say Client Hello did not receive response, it sounds that TCP connecition establishment preceeding the TLS handshake was successfull (i.e. SYN and ACK packets were flowing both sides). Did ClientHello got at least the subsequent ACK? Did RST packet arrive?
– what IP address is used when reaching to status.dev.azure.com – confirm it is 13.107.6.183 or 2620:1ec:a92::183.
– this pattern of behavior (TCP establishment gets allowed, ClientHello does not get answered / or is RST-ed) is consistent with smart FW / NGFW blocking the connection (so AzFW would be suspect here)
– differential diagnostics: are the calls to other TLS 1.2-based services working from the machine? example
(Invoke-WebRequest -Uri https://github.com).StatusCodeHello Ondrej,
Thank you for your response, much appreciated. I added the answers to your questions inline:
– When you run the script, does it say “Failed to reach the destination. This is connectivity or DNS problem, not TLS compatibility issue.” or does it say “ISSUE FOUND: This may be TLS compatibility issue!”
-> Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to 13.107.6.183:443. Error: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
– regarding the network trace. Since you say Client Hello did not receive response, it sounds that TCP connecition establishment preceeding the TLS handshake was successfull (i.e. SYN and ACK packets were flowing both sides). Did ClientHello got at least the subsequent ACK? Did RST packet arrive?
-> Yes, first 3 packages are SYN/ACK packages, then a TLSv1.2 client hello followed by 9 TCP Retransmission packages and finally a [RST, ACK, CWR] package.
– what IP address is used when reaching to status.dev.azure.com – confirm it is 13.107.6.183 or 2620:1ec:a92::183.
-> 13.107.6.183
– this pattern of behavior (TCP establishment gets allowed, ClientHello does not get answered / or is RST-ed) is consistent with smart FW / NGFW blocking the connection (so AzFW would be suspect here)
-> the weird thing is that the Windows 2022 virtual machine in the same subnet works.
– differential diagnostics: are the calls to other TLS 1.2-based services working from the machine? example (Invoke-WebRequest -Uri https://github.com).StatusCode
-> same error: Invoke-WebRequest : The underlying connection was closed: An unexpected error occurred on a send.
Thanks
Hi D.K. sorry to see you still face the issue.
The packet sequence definitelly indicates middleman (read: firewall) cutting the connection based on what it sees in ClientHello. I would focus investigation this way.
The paradox you see with WS 2022 is a good clue. What WS 2022 may do differently than your WS 2k12 VMs:
– it may favor IPv6 when resolving the DNS names and thus other set of FW rules may kick-in.
– WS 2k12 may not be including SNI in ClientHello packets (can be determined from wireshark inspection of ClientHello: “Extension: server_name”/”Server Name Indication extension” subtree present). If so, it would explain the issue since AzFW requires SNI present, see the known issues list.
Thank you Ondrej, we’ll investigate further 🙂
Is there a target date for when the rollout be completed? I.E when will TLS 1.1 & 1.0 no longer be supported
dev.azure.com has been completed now. If you’re using this domain, then you should be on TLS 1.2. visualstudio.com is currently being worked on.
Mark, I think your “TLS powershell check” should also include -UseBasicParsing. Without it, I got the following error:
Which I was unable to resolve. But thanks to https://stackoverflow.com/a/38054505/2246411 I found that the following is a better option:
I also found that the following pipeline task was much shorter (and I believe it’s equivalent):
steps: - powershell: '"TLS 1.2 readiness check:"; (Invoke-WebRequest -Uri status.dev.azure.com -UseBasicParsing).StatusDescription' failOnStderr: trueThanks, Derek for the comments on the Invote-WebRequest statement. I’ll add it to the blog post, as anything to help save people time when looking at this is great. Much appreciated.
One of our build agents is no longer connecting. I tried to run the troubleshooting tool and Invoke-WebRequest fails but everything else is green. I ran Invoke-WebRequest against github and got a 200. Any advice?
Azure DevOps TLS 1.2 transition readiness checker v. 2022-04-14
| Probing Azure DevOps sites |
Probing: status.dev.azure.com
ISSUE FOUND: This may be TLS compatibility issue!
Probe failed when TLS-negotiating to 13.107.6.183:443. Error: Unable to read data from the transport connection: An exis
ting connection was forcibly closed by the remote host.
********************************************************************************
| Analysis of TLS 1.2 compatibility: OS |
PS Version: 5.1.14393.5066
PS Edition: Desktop
Win Build Version: 10.0.14393.0
CLR Version: 4.0.30319.42000
TLS 1.2 client usage enabled.
Running Get-TlsCipherCuite check…
All enabled TLS 1.2 cipher suites: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 TLS_E
CDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_W
ITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_
AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_
CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WIT
H_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS
_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_RC4_12
8_SHA TLS_RSA_WITH_RC4_128_MD5 TLS_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_NULL_SHA TLS_PSK_WITH_AES_128_GCM_SHA256 TLS_PSK_WI
TH_AES_256_CBC_SHA384 TLS_PSK_WITH_AES_128_CBC_SHA256 TLS_PSK_WITH_NULL_SHA384 TLS_PSK_WITH_NULL_SHA256
Matching cipher suites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES
_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
At least one of the TLS 1.2 cipher suites required by Azure DevOps enabled on the machine.
Running DHE check…
Diffie-Helman key exchange disabled. Enabled cipher suites after filtering: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_EC
DHE_RSA_WITH_AES_128_GCM_SHA256
Running Group Policy check…
Group Policy cipher suites override defined: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SH
A256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS
DHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECD
HE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDS
A_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CB
C_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES
256_CBC_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_3D
ES_EDE_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA TLS_RSA_WITH_RC4_128_SHA TLS_RSA_WITH_RC4_128_MD5 TLS
_RSA_WITH_NULL_SHA256 TLS_RSA_WITH_NULL_SHA TLS_PSK_WITH_AES_256_GCM_SHA384 TLS_PSK_WITH_AES_128_GCM_SHA256 TLS_PSK_WITH
_AES_256_CBC_SHA384 TLS_PSK_WITH_AES_128_CBC_SHA256 TLS_PSK_WITH_NULL_SHA384 TLS_PSK_WITH_NULL_SHA256
Missing cipher suites:
No need to change the GP override since cipher suites required by Azure DevOps are already enabled.
| Analysis of TLS 1.2 compatibility: .NET Framework |
.NET Framework release is 4.7+ (release 528049)
TLS 1.2 enforced for applications targetting .NET Framework 4.0/4.5.x
TLS 1.2 enforced for applications targetting .NET Framework 4.0/4.5.x (32bit app on 64bit OS)
TLS 1.2 enforced for applications targetting .NET Framework 3.5
TLS 1.2 enforced for applications targetting .NET Framework 3.5 (32bit app on 64bit OS)
All mitigations required to ensure TLS 1.2-compatibility of legacy .NET applications are in place.
Hi Mustafa, good to see you checked the main things.
Seeing GitHub check works well, this leads to cipher suites problem. But before we go that route, let’s rule-out .NET Framework / PS 5 having something to do with it. Can you please run the troubleshooting checker script on PowerShell 7? This will circumvent .NET Framework and run, same as Pipeline agent, on top of .NET Core.
If the PS 7 result is the same as with PS 5 then I’d focus on the fact that DH key-exchange is disabled on your machine and check if ECDHE is disabled as well. What subkeys do you have defined under HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\ ?
identical results in PS7
in that location we have a subfolder called Diffie-Hellman with the keys Default:Not Set and Enabled:0
thanks.
There seems to be some problem that prevents usage of ECDHE cipher suites despite they are enabled at usual places. (This can be confirmed by capturing TLS handshake using Wireshark and looking at ClientHello packet to verify if cipher suites
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256are missing in client’s list.)As a workaround to ECDHE problem you can try to enable DHE cipher suites which you have configured as disabled (“Diffie-Helman key exchange disabled.”). Set
HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman!Enabledto 0xFFFFFFFF (DWORD), Restart the computer and verify the Pipeline agent and/or probe of troubleshooting checker started working.If this helps, please run
gpupdate /forceand check if your registry change persisted. If it got back toEnabled=0, it means that the change is enforced by Domain Controller and to make it permanent it needs to be configured by GPO.Keep in mind that this DHE enablement is only workaround and longer term solution is to find why ECDHE ciphers don’t work despite being enabled. Also, your security team may have reasons why they disabled DHE key-exchange so please consult this change.
The ECDHE problem may be caused by missing elliptic curves. Can you post here output of
Get-TlsEccCurve?Hello Ondrej,
Trying to fix the DHE caused the server to go into an unstable state (guessing the group policy didnt like what I did). I’m waiting for the server to be restored so I can try next steps.
Thanks for all your help so far, I’ll reply back once I’m able to run Get-TlsEccCurve
Hello Ondrej,
See below for results
PS C:\Windows\System32> Get-TlsEccCurve
curve25519
NistP256
NistP384
Thanks Mustafa. The elliptic curves NistP256 and NistP384 are ones used by AzDO, so this looks good. I admit having no further ideas why ECDHE does not work in your setup. I advise you to reach out to Azure support to get more personalized help.
Regarding Diffie-Hellman disablement: if this is security motivated, then the recommended practice known to me is not to disable the Diffie-Hellman as such, but to enforce minimum server key length to at least 2048bits. Such configuration would be compatible with Azure DevOps.
https://docs.microsoft.com/en-us/security-updates/SecurityAdvisories/2016/3174644
https://support.microsoft.com/en-us/topic/microsoft-security-advisory-updated-support-for-diffie-hellman-key-exchange-f0ad89ce-dcd5-56e2-9cee-4cbb01b4da1e
Was a final update to dev.azure.com done in the first week of May? At that time we noticed a marketplace extension that stopped functioning properly. The developer has since fixed the issue, but we need to go back and review data from over 30 projects to fix the data. Having a roll out date from that week, if one is available, would help greatly.
Hello Sarah, the global rollout of the final update to dev.azure.com took place at about 2022-05-03 14:36 UTC. Hope it helps!
Thank you, it does!!
It looks like https://orgname.visualstudio.com is still accepting TLS 1.0 & 1.1.
Is there a date when rollout will be completed?
Many of my users still use old URL form interchangeably or even preferentially … even after org settings long updated to use new URL.
Matt, this is correct – at the moment, IPv4 endpoints of visualstudio.com are the last ones remaining.
We roll this process slowly. After each step we’re closely monitoring health of the customers with higher legacy TLS exposure. Once visualstudio.com is switched, we will update here.
good to know…