With this patch cycle, we are releasing fixes that impact our self-hosted product, Azure DevOps Server, as well as Team Foundation Server 2018.3.2. Please see the release notes for additional installation instructions.
The following will be fixed with this patch:
Upgraded search plugins to use log4j core version 2.17.1.
2/23 Update: If Azure DevOps Server/TFS and Elasticsearch are installed on different machines, follow the steps outlined below.
-
Upgrade the server with the latest patch.
-
Check the registry value at HKLM:\Software\Elasticsearch\Version on the ElasticSearch server machine. Update the registry value to 5.4.1, this value is irrespective if you have higher version of ElasticSearch. If the registry value is not there, add a string value and set the Version to 5.4.1 (Name = Version, Value = 5.4.1). You will have to reindex the collection after the patch if there was no registry value.
-
Copy the content of the folder named zip, located on C:\Program Files{TFS Version Folder}\Search\zip to the Elasticsearch remote file folder.
-
Run Configure-TFSSearch.ps1 -Operation update on the Elasticsearch server machine.
2/17 Update: Do not update Log4 jar file manually to 2.17.1, this is unsupported and will cause Elasticsearch to not work properly or break.
2/8 Update: The addition of the plugins upgrading to 2.17.1 was incorrect and we apologize for the confusion that this caused. Currently, using Configure-TFS is the only supported means to update Elasticsearch. We are in the process of validating the patches on a more comprehensive configuration matrix including installation of Elasticsearch on a separate machine and will update the release notes and/or patches in the next few days.
- Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.
- Email notifications were not sent when using the @mention control in a work item.
- Preferred email address was not getting updated in user profile.
- Header was not shown in the Project Summary page.
- Improvement to Active Directory user sync.
Azure DevOps Server 2020.1.1 Patch 4
If you have Azure DevOps Server 2020.1.1, you should install Azure DevOps Server 2020.1.1 Patch 4. Check out the release notes for more details.
Verifying Installation
-
Option 1: Run
devops2020.1.1patch4.exe CheckInstall
, devops2020.1.1patch4.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed. -
Option 2: Check the version of the following file:
[INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll
. Azure DevOps Server 2020.1.1 is installed toc:\Program Files\Azure DevOps Server 2020
by default. After installing Azure DevOps Server 2020.1.1 Patch 4, the version will be 18.181.32118.5.
Azure DevOps Server 2020.0.1 Patch 9
If you have Azure DevOps Server 2020.0.1, you should install Azure DevOps Server 2020.0.1 Patch 9. Check out the release notes for more details.
Verifying Installation
-
Option 1: Run
devops2020.0.1patch9.exe CheckInstall
, devops2020.0.1patch9.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed. -
Option 2: Check the version of the following file:
[INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll
. Azure DevOps Server 2020.0.1 is installed toc:\Program Files\Azure DevOps Server 2020
by default. After installing Azure DevOps Server 2020.0.1 Patch 9, the version will be 18.17032118.4.
Azure DevOps Server 2019.1.1 Patch 13
If you have Azure DevOps Server 2019 Update 1.1, you should install Azure DevOps Server 2019 Update 1.1 Patch 13. Check out the release notes for more details
Verifying Installation
-
Option 1: Run
devops2019.1.1patch13.exe CheckInstall
.devops2019.1.1patch13.exe
is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed. -
Option 2: Check the version of the following file:
[INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll
. Azure DevOps Server 2019 is installed toc:\Program Files\Azure DevOps Server 2019
by default. After installing Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3.
Azure DevOps Server 2019.0.1 Patch 12
If you have Azure DevOps Server 2019, you should first update to Azure DevOps Server 2019.0.1. Once on 2019.0.1, install Azure DevOps Server 2019.0.1 Patch 12. Check out the release notes for more details.
Verifying Installation
-
Option 1: Run
devops2019.0.1patch12.exe CheckInstall
.devops2019.1.1patch12.exe
is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed. -
Option 2: Check the version of the following file:
[INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll
. Azure DevOps Server 2019 is installed toc:\Program Files\Azure DevOps Server 2019
by default. After installing Azure DevOps Server 2019.0.1 Patch 12, the version will be 17.143.32118.2.
TFS 2018 Update 3.2 Patch 16
If you have TFS 2018 Update 2 or Update 3, you should first update to TFS 2018 Update 3.2. Once on Update 3.2, install TFS 2018 Update 3.2 Patch 16. Check out the release notes for more details
Verifying Installation
-
Option 1: Run
tfs2018.3.2patch16.exe CheckInstall
, tfs2018.3.2patch16.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed. -
Option 2: Check the version of the following file:
[TFS_INSTALL_DIR]\Application Tier\Web Services\bin\Microsoft.TeamFoundation.WorkItemTracking.Web.dll
. TFS 2018 is installed toc:\Program Files\Microsoft Team Foundation Server 2018
by default. After installing TFS 2018 Update 3.2 Patch 16, the version will be 16.131.32118.1.
Last updated: 2/23/2022 @ 6:25 pm PST
Hi Gloridel,
We are running a single server Azure DevOps 2020 - 18.181.31626.1 (Azure DevOps Server 2020 Update 1.1)
Update 4 was applied, however our M365 Defender report shows the vulnerability against the files below:
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-1.2-api-2.9.1.jar
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar
Please advice if the M365 Defender report is to be ignored? or has the Update 4 not been applied properly?
Thanks.
In case anyone has a problem and you run into this error where it thinks your NOT running 64 Bit Java.. I noticed that with IBM Semeru java -version returns the text "64-Bit" .. instead of the "64-bit" that the ./modules/JavaHelper.psm1 is expecting around line 44. You may need to comment out accordingly.. assuming you're sure you are indeed running 64bit Java:
<code>
I think Powershell's -contains operator should be case insensitive by default but perhaps...
Could you please help, it is confusing if remediation for Log4J is in place. I followed the latest instructions for patch "Azure DevOps Server 2019.1.1 Patch 13" and it has been installed; I can verify "Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3."
Though the "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib" directory still has same Log4J version as "log4j-core-2.9.1". My expectation was it will updated to newer version.
Thanks!
“Patch can be installed” CheckInstall option reports that the patch is not installed! Any fixes for this?
Checking SOFTWARE\Microsoft\TeamFoundationServer\18.0 to see if Azure DevOps Server is installed
Found InstallPath: C:\Program Files\Azure DevOps Server 2020\
Found InstallVersion: 18.181.31230.2
Latest patch installed on machine is version 18.181.31609.2
Patch 18.181.32118.5 is the same or later version as the patch installed on machine, patch can be installed.
The Application Tier is configured.
The Search Tier is configured.
The Proxy Tier...
Hello, we currently run TFS 15.0 on prem and our vulnerability scanner has flagged this path (C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-2.4.1\lib\log4j-1.2.17.jar) for the new critical log4j vulnerability (CVE-2022-23307). Are you able to confirm if any remediation steps need to be taken for this new one as well as the original CVE-2021-44228 (or any log4j related vulnerabilities)? From what I am reading, TFS 15 is not affected by the original vulnerability but nothing on the...
Hello Gloridel,
I have a question not directly about the update but about Azure DevOps Server in general.
I think they are sitting at the right source :-)
Will in one of the next versions of Azure DevOps Server the XAML feature be taken out?
Are there any thoughts on this already, though no concrete timelines?
We are currently still using it and so I would want to know if it is being considered. It...
I second this question.
Hello Jensen,
thanks for your support. It seems, there is no clear plan available yet….
I just ran the 2019 patch 13. The patch did not update elasticsearch on the remote server. We have 2 AzureDevops 2019 servers and 1 server running elasticsearch. When I go to the search/zip folder on the AzureDevOps server and run .\Configure-TFSSearch.ps1 -Operation update , I get [ERROR]: Elasticsearch not found.
Hi Payne, I have updated the blog post with a note about installations on multiple servers.
Hi Payne, we are in the process of validating this configuration and will update this blog post with results.
Hi, because there is no fix for this vunlnerability, from Microsoft, any time soon, I will recommend to our team that we uninstall elastic search in it’s current configuration.
Same here, I highly doubt that a 2.17.1 version of log4J was shipped.
First I updated ES, but saw a "C:\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar"
So I removed and installed again, and still this log4j-core-2.9.1.jar
I suspected the source in C:\Program Files\Azure DevOps Server 2020\Search\zip but all is dated from today 02/03/22 so it seems to be legit
So it seems that there's a bug in this patch.
<code>
Hi Vincent, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.
I have the same question. Is 2.9.1 version used in DevOps 2020.1.1 patch 4 considered to be safe?
I have applied the patch and according to the process log for the patch it was successful. We still have the log4j-core-2.9.1.jar file. It seems it's a new copy of the file, it's a litle bit less in size than the log4j-core-2.9.1.jar on the system before the patch4.
Our vulnerability scanner complains about this file and says JNDI Class: Found.
Should we consider this log4j-core-2.9.1.jar file to be safe?
Hi Gloridel!
We installed patch 9 for Azure DevOps Server 2020.01. and then ran "Configure-TFSSearch.ps1 -Operation update".
It worked that the jndilookup class was removed from the log4j binaries (see release notes "Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.").
What didn't work is the release note "Upgraded search plugins to use log4j core version 2.17.1.". After installing the patch and reconfiguring the search, the old version 2.9.1 is still present. Even if you look...
Hi Florian, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.
After updating Azure devops/Alm search on a separate search server with the patch that comes with Azure DevOps Server 2020.1.1 Patch 4, we were unable to (re-)configure the search feature in the Azure Devops Management console. There were 2 seperate errors regarding plugins:
<code>
and
<code>
Eventually we managed to get the search server and feature working by renaming 2 jar files and descriptors in contained in the almsearchauthpluginv6.2 and almsearchv6.2 zip files that comes with the installation...
Edwin,
Thank you we hit this and your post was a big help!
Hi Edwin, As you are using two different servers for AzureDevops and Search, you have to replace Search server files with zip files of AzureDevops Server which you get after applying the patch before configuring the search.
Hello Kumar,
thats exactly what i did.
Could you please be a little more detailed? I don’t understand what you mean.
Hi Edwin, please use the steps as listed in the release notes and let us know if you continue have issues.
@Gloridel
please advise how to fix the problem –
search plugin with version: ‘6.2.4.22’ is not compatible, supported search plugin version: 6.2.4.18
We have the exact same problem… Your solution doesn’t really appeal to me – is there a new patch coming for this perhaps? @Gloridel?
Hi Staffan, for now we don’t have additional plans to address this in future patches. I will continue to post updates to this blog post with results from additional validations the team is performing.
Hi Edwin Siebes !
Can you provide more details how to fix it –
Thanks!
The quick and dirty way to fix it, was to (before install) change all the references in the modules with version 6.2.4.22 to 6.2.4.18 and also rename the jar files to that version. (I’ve taken the files that come with the patched azure devops instance.
After that run the upgrade.