Updated: Azure DevOps Server and Team Foundation Server patches

Gloridel Morales

Senior Technical Program Manager

With this patch cycle, we are releasing fixes that impact our self-hosted product, Azure DevOps Server, as well as Team Foundation Server 2018.3.2. Please see the release notes for additional installation instructions.

The following will be fixed with this patch:

~~Upgraded search plugins to use log4j core version 2.17.1~~.

2/23 Update: If Azure DevOps Server/TFS and Elasticsearch are installed on different machines, follow the steps outlined below.

Upgrade the server with the latest patch.
Check the registry value at HKLM:\Software\Elasticsearch\Version on the ElasticSearch server machine. Update the registry value to 5.4.1, this value is irrespective if you have higher version of ElasticSearch. If the registry value is not there, add a string value and set the Version to 5.4.1 (Name = Version, Value = 5.4.1). You will have to reindex the collection after the patch if there was no registry value.
Copy the content of the folder named zip, located on C:\Program Files{TFS Version Folder}\Search\zip to the Elasticsearch remote file folder.
Run Configure-TFSSearch.ps1 -Operation update on the Elasticsearch server machine.

2/17 Update: Do not update Log4 jar file manually to 2.17.1, this is unsupported and will cause Elasticsearch to not work properly or break.

2/8 Update: The addition of the plugins upgrading to 2.17.1 was incorrect and we apologize for the confusion that this caused. Currently, using Configure-TFS is the only supported means to update Elasticsearch. We are in the process of validating the patches on a more comprehensive configuration matrix including installation of Elasticsearch on a separate machine and will update the release notes and/or patches in the next few days.

Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.
Email notifications were not sent when using the @mention control in a work item.
Preferred email address was not getting updated in user profile.
Header was not shown in the Project Summary page.
Improvement to Active Directory user sync.

Azure DevOps Server 2020.1.1 Patch 4

If you have Azure DevOps Server 2020.1.1, you should install Azure DevOps Server 2020.1.1 Patch 4. Check out the release notes for more details.

Verifying Installation

Option 1: Run devops2020.1.1patch4.exe CheckInstall, devops2020.1.1patch4.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.
Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll. Azure DevOps Server 2020.1.1 is installed to c:\Program Files\Azure DevOps Server 2020 by default. After installing Azure DevOps Server 2020.1.1 Patch 4, the version will be 18.181.32118.5.

Azure DevOps Server 2020.0.1 Patch 9

If you have Azure DevOps Server 2020.0.1, you should install Azure DevOps Server 2020.0.1 Patch 9. Check out the release notes for more details.

Verifying Installation

Option 1: Run devops2020.0.1patch9.exe CheckInstall, devops2020.0.1patch9.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.
Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll. Azure DevOps Server 2020.0.1 is installed to c:\Program Files\Azure DevOps Server 2020 by default. After installing Azure DevOps Server 2020.0.1 Patch 9, the version will be 18.17032118.4.

Azure DevOps Server 2019.1.1 Patch 13

If you have Azure DevOps Server 2019 Update 1.1, you should install Azure DevOps Server 2019 Update 1.1 Patch 13. Check out the release notes for more details

Verifying Installation

Option 1: Run devops2019.1.1patch13.exe CheckInstall. devops2019.1.1patch13.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.
Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll. Azure DevOps Server 2019 is installed to c:\Program Files\Azure DevOps Server 2019 by default. After installing Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3.

Azure DevOps Server 2019.0.1 Patch 12

If you have Azure DevOps Server 2019, you should first update to Azure DevOps Server 2019.0.1. Once on 2019.0.1, install Azure DevOps Server 2019.0.1 Patch 12. Check out the release notes for more details.

Verifying Installation

Option 1: Run devops2019.0.1patch12.exe CheckInstall. devops2019.1.1patch12.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.
Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll. Azure DevOps Server 2019 is installed to c:\Program Files\Azure DevOps Server 2019 by default. After installing Azure DevOps Server 2019.0.1 Patch 12, the version will be 17.143.32118.2.

TFS 2018 Update 3.2 Patch 16

If you have TFS 2018 Update 2 or Update 3, you should first update to TFS 2018 Update 3.2. Once on Update 3.2, install TFS 2018 Update 3.2 Patch 16. Check out the release notes for more details

Verifying Installation

Option 1: Run tfs2018.3.2patch16.exe CheckInstall, tfs2018.3.2patch16.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.
Option 2: Check the version of the following file: [TFS_INSTALL_DIR]\Application Tier\Web Services\bin\Microsoft.TeamFoundation.WorkItemTracking.Web.dll. TFS 2018 is installed to c:\Program Files\Microsoft Team Foundation Server 2018 by default. After installing TFS 2018 Update 3.2 Patch 16, the version will be 16.131.32118.1.

Last updated: 2/23/2022 @ 6:25 pm PST

Author

Gloridel Morales

Senior Technical Program Manager

Gloridel is a Senior Technical Program Manager on the Azure DevOps team.

45 comments

Discussion is closed. Login to edit/delete existing comments.

Sort by :

Newest

Newest Popular Oldest

Virantha T March 9, 2022 0

Hi Gloridel,

We are running a single server Azure DevOps 2020 - 18.181.31626.1 (Azure DevOps Server 2020 Update 1.1)
Update 4 was applied, however our M365 Defender report shows the vulnerability against the files below:
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-1.2-api-2.9.1.jar
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar

Please advice if the M365 Defender report is to be ignored? or has the Update 4 not been applied properly?

Thanks.

Read more
Hi Gloridel,

We are running a single server Azure DevOps 2020 – 18.181.31626.1 (Azure DevOps Server 2020 Update 1.1)
Update 4 was applied, however our M365 Defender report shows the vulnerability against the files below:
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-1.2-api-2.9.1.jar
C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar

Please advice if the M365 Defender report is to be ignored? or has the Update 4 not been applied properly?

Thanks.

Read less
Andrew Kanieski March 5, 2022 · Edited 0
In case anyone has a problem and you run into this error where it thinks your NOT running 64 Bit Java.. I noticed that with IBM Semeru java -version returns the text "64-Bit" .. instead of the "64-bit" that the ./modules/JavaHelper.psm1 is expecting around line 44. You may need to comment out accordingly.. assuming you're sure you are indeed running 64bit Java:

<code>

I think Powershell's -contains operator should be case insensitive by default but perhaps...
Read more
In case anyone has a problem and you run into this error where it thinks your NOT running 64 Bit Java.. I noticed that with IBM Semeru java -version returns the text “64-Bit” .. instead of the “64-bit” that the ./modules/JavaHelper.psm1 is expecting around line 44. You may need to comment out accordingly.. assuming you’re sure you are indeed running 64bit Java:
```
            #if ($words -contains "64-bit")
            #{
                $ret.Is64Bit = $true
            #}
```
I think Powershell’s -contains operator should be case insensitive by default but perhaps there is some bad character hidden in there.. but commenting out that if check allowed us to continue.
Read less
Kamlesh Kumawat February 28, 2022 0

Could you please help, it is confusing if remediation for Log4J is in place. I followed the latest instructions for patch "Azure DevOps Server 2019.1.1 Patch 13" and it has been installed; I can verify "Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3."

Though the "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib" directory still has same Log4J version as "log4j-core-2.9.1". My expectation was it will updated to newer version.

Thanks!

Read more
Could you please help, it is confusing if remediation for Log4J is in place. I followed the latest instructions for patch “Azure DevOps Server 2019.1.1 Patch 13” and it has been installed; I can verify “Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3.”

Though the “C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib” directory still has same Log4J version as “log4j-core-2.9.1”. My expectation was it will updated to newer version.

Thanks!

Read less
Franz Haas February 25, 2022 0

“Patch can be installed” CheckInstall option reports that the patch is not installed! Any fixes for this?

Checking SOFTWARE\Microsoft\TeamFoundationServer\18.0 to see if Azure DevOps Server is installed
Found InstallPath: C:\Program Files\Azure DevOps Server 2020\
Found InstallVersion: 18.181.31230.2
Latest patch installed on machine is version 18.181.31609.2
Patch 18.181.32118.5 is the same or later version as the patch installed on machine, patch can be installed.
The Application Tier is configured.
The Search Tier is configured.
The Proxy Tier...
Read more
“Patch can be installed” CheckInstall option reports that the patch is not installed! Any fixes for this?

Checking SOFTWARE\Microsoft\TeamFoundationServer\18.0 to see if Azure DevOps Server is installed
Found InstallPath: C:\Program Files\Azure DevOps Server 2020\
Found InstallVersion: 18.181.31230.2
Latest patch installed on machine is version 18.181.31609.2
Patch 18.181.32118.5 is the same or later version as the patch installed on machine, patch can be installed.
The Application Tier is configured.
The Search Tier is configured.
The Proxy Tier is not configured.
This patch does not apply to Azure DevOps Server version 18.181.31230.2.

Read less
techatbc February 18, 2022 0

Hello, we currently run TFS 15.0 on prem and our vulnerability scanner has flagged this path (C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-2.4.1\lib\log4j-1.2.17.jar) for the new critical log4j vulnerability (CVE-2022-23307). Are you able to confirm if any remediation steps need to be taken for this new one as well as the original CVE-2021-44228 (or any log4j related vulnerabilities)? From what I am reading, TFS 15 is not affected by the original vulnerability but nothing on the...
Read more
Hello, we currently run TFS 15.0 on prem and our vulnerability scanner has flagged this path (C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-2.4.1\lib\log4j-1.2.17.jar) for the new critical log4j vulnerability (CVE-2022-23307). Are you able to confirm if any remediation steps need to be taken for this new one as well as the original CVE-2021-44228 (or any log4j related vulnerabilities)? From what I am reading, TFS 15 is not affected by the original vulnerability but nothing on the new one. Thanks a lot!

Read less
Muehlberger, Rainer February 16, 2022 0

Hello Gloridel,
I have a question not directly about the update but about Azure DevOps Server in general.
I think they are sitting at the right source :-)
Will in one of the next versions of Azure DevOps Server the XAML feature be taken out?
Are there any thoughts on this already, though no concrete timelines?
We are currently still using it and so I would want to know if it is being considered. It...
Read more
Hello Gloridel,
I have a question not directly about the update but about Azure DevOps Server in general.
I think they are sitting at the right source 🙂
Will in one of the next versions of Azure DevOps Server the XAML feature be taken out?
Are there any thoughts on this already, though no concrete timelines?
We are currently still using it and so I would want to know if it is being considered. It would help us prepare accordingly.

Thank you very much.

Read less
- Tore Østergaard Jensen (TORE) February 18, 2022 0
  
  I second this question.
  - Muehlberger, Rainer February 22, 2022 0
    
    Hello Jensen,
    thanks for your support. It seems, there is no clear plan available yet….
Payne, Scott February 3, 2022 0

I just ran the 2019 patch 13. The patch did not update elasticsearch on the remote server. We have 2 AzureDevops 2019 servers and 1 server running elasticsearch. When I go to the search/zip folder on the AzureDevOps server and run .\Configure-TFSSearch.ps1 -Operation update , I get [ERROR]: Elasticsearch not found.
- Gloridel Morales Author February 23, 2022 0
  
  Hi Payne, I have updated the blog post with a note about installations on multiple servers.
- Gloridel Morales Author February 8, 2022 0
  
  Hi Payne, we are in the process of validating this configuration and will update this blog post with results.
  - Payne, Scott February 23, 2022 · Edited 0
    
    Hi, because there is no fix for this vunlnerability, from Microsoft, any time soon, I will recommend to our team that we uninstall elastic search in it’s current configuration.
Vincent Tollu February 3, 2022 · Edited 0
Same here, I highly doubt that a 2.17.1 version of log4J was shipped.

First I updated ES, but saw a "C:\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar"

So I removed and installed again, and still this log4j-core-2.9.1.jar

I suspected the source in C:\Program Files\Azure DevOps Server 2020\Search\zip but all is dated from today 02/03/22 so it seems to be legit

So it seems that there's a bug in this patch.

<code>

Read more
Same here, I highly doubt that a 2.17.1 version of log4J was shipped.

First I updated ES, but saw a “C:\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar”

So I removed and installed again, and still this log4j-core-2.9.1.jar

I suspected the source in C:\Program Files\Azure DevOps Server 2020\Search\zip but all is dated from today 02/03/22 so it seems to be legit

So it seems that there’s a bug in this patch.
```
    Directory: C:\Program Files\Azure DevOps Server 2020\Search\zip


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       03/02/2022     10:55                modules
-a----       03/02/2022     10:55           7250 almsearchauthpluginv6.2.zip
-a----       03/02/2022     10:55       11206212 almsearchicupluginv6.2.zip
-a----       03/02/2022     10:55         204738 almsearchv6.2.zip
-a----       26/08/2021     11:37          20545 Configure-TFSSearch.ps1
-a----       03/02/2022     10:55       29038876 elasticsearchv6.2.zip
-a----       04/10/2020     22:02           5225 log4j2.properties
-a----       28/07/2021     22:02           4793 README.txt
-a----       28/05/2020     22:02          14313 relevance.xml
```
Read less
- Gloridel Morales Author February 8, 2022 0
  
  Hi Vincent, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.
  - Jan Slansky March 11, 2022 0
    
    I have the same question. Is 2.9.1 version used in DevOps 2020.1.1 patch 4 considered to be safe?
  - Johannes Karl Karlsson March 10, 2022 0
    
    I have applied the patch and according to the process log for the patch it was successful. We still have the log4j-core-2.9.1.jar file. It seems it's a new copy of the file, it's a litle bit less in size than the log4j-core-2.9.1.jar on the system before the patch4.
    
    Our vulnerability scanner complains about this file and says JNDI Class: Found.
    
    Should we consider this log4j-core-2.9.1.jar file to be safe?
    
    Read more
    I have applied the patch and according to the process log for the patch it was successful. We still have the log4j-core-2.9.1.jar file. It seems it’s a new copy of the file, it’s a litle bit less in size than the log4j-core-2.9.1.jar on the system before the patch4.
    
    Our vulnerability scanner complains about this file and says JNDI Class: Found.
    
    Should we consider this log4j-core-2.9.1.jar file to be safe?
    
    Read less
Florian Lauger February 3, 2022 0

Hi Gloridel!

We installed patch 9 for Azure DevOps Server 2020.01. and then ran "Configure-TFSSearch.ps1 -Operation update".

It worked that the jndilookup class was removed from the log4j binaries (see release notes "Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.").

What didn't work is the release note "Upgraded search plugins to use log4j core version 2.17.1.". After installing the patch and reconfiguring the search, the old version 2.9.1 is still present. Even if you look...
Read more
Hi Gloridel!

We installed patch 9 for Azure DevOps Server 2020.01. and then ran “Configure-TFSSearch.ps1 -Operation update”.

It worked that the jndilookup class was removed from the log4j binaries (see release notes “Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.”).

What didn’t work is the release note “Upgraded search plugins to use log4j core version 2.17.1.”. After installing the patch and reconfiguring the search, the old version 2.9.1 is still present. Even if you look into the patch (ZIP), you can see that the old version is still delivered.

Is this expected or is the patch not complete/buggy?

Read less
- Gloridel Morales Author February 8, 2022 0
  
  Hi Florian, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.
edwin siebes February 1, 2022 0
After updating Azure devops/Alm search on a separate search server with the patch that comes with Azure DevOps Server 2020.1.1 Patch 4, we were unable to (re-)configure the search feature in the Azure Devops Management console. There were 2 seperate errors regarding plugins:

<code>

and

<code>

Eventually we managed to get the search server and feature working by renaming 2 jar files and descriptors in contained in the almsearchauthpluginv6.2 and almsearchv6.2 zip files that comes with the installation...
Read more
After updating Azure devops/Alm search on a separate search server with the patch that comes with Azure DevOps Server 2020.1.1 Patch 4, we were unable to (re-)configure the search feature in the Azure Devops Management console. There were 2 seperate errors regarding plugins:
```
 search plugin with version: '6.2.4.22' is not compatible, supported search plugin version: 6.2.4.18
[Error  @16:55:10.884] Elasticsearch plugin: 'AlmSearchPlugin' with version: '6.2.4.22' is not supported. Ensure that your remote Search server has been updated to the latest version prior to configuring Search.
```
and
```
[Error  @09:19:50.707] search auth plugin with version: '6.2.4.8' is not compatible, supported search auth plugin version: 6.2.4.4
[Info ] Node returned: Error
[Error  ] Elasticsearch plugin: 'AlmSearchAuthPlugin' with version: '6.2.4.8' is not supported. Ensure that your remote Search server has been updated to the latest version prior to configuring Search.
```
Eventually we managed to get the search server and feature working by renaming 2 jar files and descriptors in contained in the almsearchauthpluginv6.2 and almsearchv6.2 zip files that comes with the installation to different versions, plus a powershell properties file in the containing modules folder.
After a new installation and reindexing of the elastic search database everything worked.

This was a quick and dirty solution. Is there a better way to get this to work? And also, was this unique to our installation of Azure devops, or could this happen more often?
Read less
- Vito A Lodese Jr March 5, 2022 0
  
  Edwin,
  
  Thank you we hit this and your post was a big help!
- Kumar Shanu (MINDTREE LIMITED) February 9, 2022 0
  
  Hi Edwin, As you are using two different servers for AzureDevops and Search, you have to replace Search server files with zip files of AzureDevops Server which you get after applying the patch before configuring the search.
  - edwin siebes February 17, 2022 0
    
    Hello Kumar,
    thats exactly what i did.
  - Staffan Westin February 9, 2022 0
    
    Could you please be a little more detailed? I don’t understand what you mean.
- Gloridel Morales Author February 8, 2022 0
  
  Hi Edwin, please use the steps as listed in the release notes and let us know if you continue have issues.
  - Владимир Конев February 8, 2022 0
    
    @Gloridel
    please advise how to fix the problem –
    search plugin with version: ‘6.2.4.22’ is not compatible, supported search plugin version: 6.2.4.18
- Staffan Westin February 7, 2022 0
  
  We have the exact same problem… Your solution doesn’t really appeal to me – is there a new patch coming for this perhaps? @Gloridel?
  - Gloridel Morales Author February 8, 2022 · Edited 0
    
    Hi Staffan, for now we don’t have additional plans to address this in future patches. I will continue to post updates to this blog post with results from additional validations the team is performing.
- Владимир Конев February 4, 2022 0
  Hi Edwin Siebes !
  Can you provide more details how to fix it –
```
search plugin with version: '6.2.4.22' is not compatible, supported search plugin version: 6.2.4.18
```
  Thanks!
  - edwin siebes February 17, 2022 0
    
    The quick and dirty way to fix it, was to (before install) change all the references in the modules with version 6.2.4.22 to 6.2.4.18 and also rename the jar files to that version. (I’ve taken the files that come with the patched azure devops instance.
    After that run the upgrade.