Updated: Azure DevOps (and Azure DevOps Server) and the log4j vulnerability
Gloridel Morales
Update: We released patches for Azure DevOps Server and TFS 2018.3.2 to include an upgraded version of Elasticsearch. Check out the blog post for details.
For the most part, Azure DevOps (and Azure DevOps Server) are built on .NET and do not use the Apache log4j library whose vulnerabilities (CVE-2021-44228, CVE-2021-45046, Microsoft security blog post) have been the focus of so much recent attention. The Search feature in both Azure DevOps and Azure DevOps Server does use this library, however, as part of its dependency on Elasticsearch.
Azure DevOps is not vulnerable. Even so, we are taking this opportunity to make additional improvements as part of our overall defense in depth strategy for the hosted service.
For Azure DevOps Server (and older versions of Team Foundation Server):
- Installations without Search configured are not vulnerable and no action is required.
-
For Azure DevOps Server 2020, Azure DevOps Server 2019, and Team Foundation Server 2018 our recommendation is the same:
-
Upgrade the Java Virtual Machine on the server where the Search feature is installed to the latest release with the same major version (and then restart Elasticsearch).
-
Remove the JndiLookup class from the jar file (and then restart Elasticsearch). Most references to this online give a command that will work only on Linux. MVP Jesse Houwing has helpfully included commands that will work on Windows using 7-zip in his blog post here. They will be similar to: 7z.exe d log4j-core-.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
-
Team Foundation Server 2017 is not vulnerable to CVE-2021-44228 or CVE-2021-45046. It is, however, dependent on a version of Elasticsearch that is end-of-life from a support perspective. We recommend upgrading to a newer version of Team Foundation Server / Azure DevOps Server (documentation) or uninstalling the Search feature (documentation).
We are still exploring generation of patches to simplify this process, and will continue posting updates here as we learn more.
Additionally, the Linux versions of the Azure Pipelines agent include an older version of log4j as part of the Team Explorer Everywhere command line used to interact with Team Foundation Version Control. This version of log4j is not vulnerable to CVE-2021-44228 or CVE-2021-45046. It is end-of-life and includes other vulnerabilities, but we have previously confirmed that these vulnerabilities are not exploitable on Azure Pipelines agents. No action is required for users of Linux Azure Pipelines agents.
Last updated: 1/26/2022 @ 12:06 pm PST
Light
Dark
9 comments
For a more detailed analysis of Azure DevOps Server and Team Foundation Server check my blogpost.
Last week we opened a ticket with premier support where we were assured that no Log4j is being used. Was it a miscommunication or lack of knowledge?
The statement is written in a way that ElasticSearch bein used with Log4J is not so much of a problem. Could you please elaborate on this?
Would be nice to get an update.
For Azure DevOps, the cloud service, everything is patched and no problem.
For Azure DevOps Server 2019 and up with Search enabled, Elastic Search 6.2.4 is deployed. Elastic Search relies on log4net and this version is vulnerable to information disclosure. According to Elastic this version should not be vulnerable to the RCE-part of this bug.
For Team Foundation Server 2018 update 2 and up and with Search enabled relies on Elastic Search 5.4.1 and by default is installed on JVM 8. This combination is fully vulnerable to the RCE element of this bug.
Both versions can’t be ‘patched’ with the JVM setting as the version of log4j is too old.
Due to the way search is deployed by default, on the same host as the application tier, bound to the local loopback interface, the attack surface is reduced. When the guidance for securing communications between application tier and stand-alone versions of Elastic Search are followed, then the communication channel should be secured by IPSEC and limited to only traffic between the application tier and elastic search.
Be mindful that Azure Pipelines can make outbound calls from the application tier when a stage is configured to be agentless and while running release gates. Those calls can probably reach Elastic Search, even if it’s installed correctly and securely.
Regardless, as this blog mentions, it’s prudent (why not just say very important) to follow the guidance from the log4j team and to remove the JNDI classes from the log4j-core-* jar files.
As an alternative, you can turn off Search and uninstall the Elastic Search service.
Versions prior to TFS 2018 update 2 seem not to be vulnerable to this bug.
Using https://github.com/hillu/local-log4j-vuln-scanner indicates that Azure Dev Ops Server 2020.1.1 is affected in the ElasticSearch components.
o indicator for vulnerable component found in c:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar (org/apache/logging/log4j/core/net/JndiManager$JndiManagerFactory.class): log4j 2.9.1-2.10.0
o indicator for vulnerable component found in c:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar (org/apache/logging/log4j/core/pattern/MessagePatternConverter.class): log4j 2.8-2.9
o indicator for vulnerable component found in c:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar (org/apache/logging/log4j/core/net/JndiManager$1.class): log4j 2.4-2.11.2
o indicator for vulnerable component found in c:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar (org/apache/logging/log4j/core/net/JndiManager.class): log4j 2.9.0-2.11.2
Does Update 3 update the ElasticSearch version to 7.16.1, log4j 2. 16,or provide any mitigations of the log4j class.
Or are we on our own to remove the jndi class?
currently you are on your own there is no patch or hotfix available from Microsoft yet to patch or upgrade Elastic Search.
Hi Gloridel,
is there already a ETA for the Elastic Search fix?
Hi Markus, we plan to release fixes with our next patch release but don’t have a date yet.
Hi Gloridel, do you have any update information about Elastic Search v6.2 fix?
Hi Oleg, we published patches to address log4j vulnerability. Take a look at the blog post for more details.