Updated: Azure DevOps Server and Team Foundation Server patches

Gloridel Morales

With this patch cycle, we are releasing fixes that impact our self-hosted product, Azure DevOps Server, as well as Team Foundation Server 2018.3.2. Please see the release notes for additional installation instructions.

The following will be fixed with this patch:

  • Upgraded search plugins to use log4j core version 2.17.1.

2/23 Update: If Azure DevOps Server/TFS and Elasticsearch are installed on different machines, follow the steps outlined below.

  1. Upgrade the server with the latest patch.

  2. Check the registry value at HKLM:\Software\Elasticsearch\Version on the ElasticSearch server machine. Update the registry value to 5.4.1, this value is irrespective if you have higher version of ElasticSearch. If the registry value is not there, add a string value and set the Version to 5.4.1 (Name = Version, Value = 5.4.1). You will have to reindex the collection after the patch if there was no registry value.

  3. Copy the content of the folder named zip, located on C:\Program Files{TFS Version Folder}\Search\zip to the Elasticsearch remote file folder.

  4. Run Configure-TFSSearch.ps1 -Operation update on the Elasticsearch server machine.

2/17 Update: Do not update Log4 jar file manually to 2.17.1, this is unsupported and will cause Elasticsearch to not work properly or break.

2/8 Update: The addition of the plugins upgrading to 2.17.1 was incorrect and we apologize for the confusion that this caused. Currently, using Configure-TFS is the only supported means to update Elasticsearch. We are in the process of validating the patches on a more comprehensive configuration matrix including installation of Elasticsearch on a separate machine and will update the release notes and/or patches in the next few days.

  • Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.
  • Email notifications were not sent when using the @mention control in a work item.
  • Preferred email address was not getting updated in user profile.
  • Header was not shown in the Project Summary page.
  • Improvement to Active Directory user sync.

Azure DevOps Server 2020.1.1 Patch 4

If you have Azure DevOps Server 2020.1.1, you should install Azure DevOps Server 2020.1.1 Patch 4. Check out the release notes for more details.

Verifying Installation

  • Option 1: Run devops2020.1.1patch4.exe CheckInstall, devops2020.1.1patch4.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.

  • Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll. Azure DevOps Server 2020.1.1 is installed to c:\Program Files\Azure DevOps Server 2020 by default. After installing Azure DevOps Server 2020.1.1 Patch 4, the version will be 18.181.32118.5.

Azure DevOps Server 2020.0.1 Patch 9

If you have Azure DevOps Server 2020.0.1, you should install Azure DevOps Server 2020.0.1 Patch 9. Check out the release notes for more details.

Verifying Installation

  • Option 1: Run devops2020.0.1patch9.exe CheckInstall, devops2020.0.1patch9.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.

  • Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2020\Application Tier\bin\Microsoft.Teamfoundation.Framework.Server.dll. Azure DevOps Server 2020.0.1 is installed to c:\Program Files\Azure DevOps Server 2020 by default. After installing Azure DevOps Server 2020.0.1 Patch 9, the version will be 18.17032118.4.

Azure DevOps Server 2019.1.1 Patch 13

If you have Azure DevOps Server 2019 Update 1.1, you should install Azure DevOps Server 2019 Update 1.1 Patch 13. Check out the release notes for more details

Verifying Installation

  • Option 1: Run devops2019.1.1patch13.exe CheckInstall. devops2019.1.1patch13.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.

  • Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll. Azure DevOps Server 2019 is installed to c:\Program Files\Azure DevOps Server 2019 by default. After installing Azure DevOps Server 2019.1.1 Patch 13, the version will be 17.153.32118.3.

Azure DevOps Server 2019.0.1 Patch 12

If you have Azure DevOps Server 2019, you should first update to Azure DevOps Server 2019.0.1. Once on 2019.0.1, install Azure DevOps Server 2019.0.1 Patch 12. Check out the release notes for more details.

Verifying Installation

  • Option 1: Run devops2019.0.1patch12.exe CheckInstall. devops2019.1.1patch12.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.

  • Option 2: Check the version of the following file: [INSTALL_DIR]\Azure DevOps Server 2019\Application Tier\Web Services\bin\Microsoft.VisualStudio.Services.Feed.Server.dll. Azure DevOps Server 2019 is installed to c:\Program Files\Azure DevOps Server 2019 by default. After installing Azure DevOps Server 2019.0.1 Patch 12, the version will be 17.143.32118.2.

TFS 2018 Update 3.2 Patch 16

If you have TFS 2018 Update 2 or Update 3, you should first update to TFS 2018 Update 3.2. Once on Update 3.2, install TFS 2018 Update 3.2 Patch 16. Check out the release notes for more details

Verifying Installation

  • Option 1: Run tfs2018.3.2patch16.exe CheckInstall, tfs2018.3.2patch16.exe is the file that is downloaded from the link above. The output of the command will either say that the patch has been installed, or that is not installed.

  • Option 2: Check the version of the following file: [TFS_INSTALL_DIR]\Application Tier\Web Services\bin\Microsoft.TeamFoundation.WorkItemTracking.Web.dll. TFS 2018 is installed to c:\Program Files\Microsoft Team Foundation Server 2018 by default. After installing TFS 2018 Update 3.2 Patch 16, the version will be 16.131.32118.1.

Last updated: 2/23/2022 @ 6:25 pm PST

45 comments

Discussion is closed. Login to edit/delete existing comments.

  • Michael Hauer 0

    Please could you be more specific about Improvement to Active Directory user sync..
    I have issues with AD group-name changes not getting synced on our 2020.1.1 Patch 2 instance. I didn’t find the time yet to investigate. Are the mentioned improvements performance related or bug fixes (or both).

    • Roland Kieslinger 0

      Yes, I have the same problem and same question.

    • Tore Østergaard Jensen (TORE) 0

      I am also wondering what lies behind this one.

      For some time the ‘Team Foundation Server Periodic Identity Synchronization’ has been the job using (by far) the most runtime of any job when looking at http://localhost:8080/tfs/_oi/_jobMonitoring?_a=summary

      • Gloridel MoralesMicrosoft employee 0

        Hi Tore, I will share your concern with the team for investigation.

        • Angel WongMicrosoft employee 0

          Hi all, the small patch referenced is setting up the groundwork for a fix to the issue you referenced here. You ought to see the full fix in the upcoming release for current versions and the upcoming patches for older versions.

    • Gloridel MoralesMicrosoft employee 0

      Hi Michael, we are working on re-releasing 2020.1.1 to address the AD sync issue. I will provide an update here as soon I have more details.

  • Jarrod Hannah 0

    Could I get some confirmation that I’m not crazy?

    “tfs2018.3.2patch16.exe ExtractZip” shows that elacsticsearchv5.zip is what is included in the patch (not a new version) and is still using Log4J 2.8.x (not 2.17.1 as stated in the release notes).

    Am I doing something wrong, have I lost my mind… or… did someone package the wrong version with the patch?

    Someone please tell me I am wrong.

    • Gloridel MoralesMicrosoft employee 0

      Hi Jarrod, we are working on updating the release notes for TFS 2018.3.2. To install the patch please follow the installation steps listed below
      1. Upgrade the server with Patch 16.
      2. Check the registry value at HKLM:\Software\Elasticsearch\Version. If the registry value is not there, add a string value and set the Version to 5.4.1 (Name = Version, Value = 5.4.1).
      3. Run the update command PS C:\Program Files{TFS Version Folder}\Search\zip> .\Configure-TFSSearch.ps1 -Operation update as provided in the readme file. It may return a warning like: Unable to connect to the remote server. Don’t close the window, as the update is performing retries until it is completed.

      • Jarrod Hannah 0

        Hi Gloridel,
        Thank you for the info and instructions.
        After running the update command, I am STILL showing Log4J-core-2.8.2 I do not see any log4j updates in any of the zip files…. I have since removed, re installed and reupdated using the “Configure-TFSSearch” script but I always end up with “Log4J-core-2.8.2.jar” please, let me know what I have done wrong, I feel like I am going crazy 😛

      • Payne, Scott 0

        I followed the steps above as well. I ran .\Configure-TFSSearch.ps1 -Operation update and it came back with Elasticsearch not found. I rean it from the devops application server that I used to configure Elasticsearch. Elasticsearch is running on a remote server that does not have devops installed.

    • Matt Hrynkow 0

      You’re not crazy Jarrod. I saw the same thing with mine but it was 2.7.1 instead. I can see where you get your confusion from.

    • Gloridel MoralesMicrosoft employee 0

      Hi Jarrod, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.

    • Gloridel MoralesMicrosoft employee 0

      Hi Joakim, we are currently testing this scenario and will provide more details as soon as we have them.

    • Travis Johnson 0

      I ran through this in a test environment today and “Configure-TFSSearch.ps1 – Operation update” using the contents of the new zip file worked fine on the external search server. The registry key value mentioned in step 2 of the instructions https://docs.microsoft.com/en-us/azure/devops/server/release-notes/azuredevops2020u1?view=azure-devops#installation-steps existed and was set to 6.2.4 before and after the update.

    • Gloridel MoralesMicrosoft employee 0

      Hi Joakim, I have updated the blog post with a note about installations on multiple servers.

  • edwin siebes 0

    After updating Azure devops/Alm search on a separate search server with the patch that comes with Azure DevOps Server 2020.1.1 Patch 4, we were unable to (re-)configure the search feature in the Azure Devops Management console. There were 2 seperate errors regarding plugins:

     search plugin with version: '6.2.4.22' is not compatible, supported search plugin version: 6.2.4.18
    [Error  @16:55:10.884] Elasticsearch plugin: 'AlmSearchPlugin' with version: '6.2.4.22' is not supported. Ensure that your remote Search server has been updated to the latest version prior to configuring Search.

    and

    [Error  @09:19:50.707] search auth plugin with version: '6.2.4.8' is not compatible, supported search auth plugin version: 6.2.4.4
    [Info ] Node returned: Error
    [Error  ] Elasticsearch plugin: 'AlmSearchAuthPlugin' with version: '6.2.4.8' is not supported. Ensure that your remote Search server has been updated to the latest version prior to configuring Search.
    

    Eventually we managed to get the search server and feature working by renaming 2 jar files and descriptors in contained in the almsearchauthpluginv6.2 and almsearchv6.2 zip files that comes with the installation to different versions, plus a powershell properties file in the containing modules folder.
    After a new installation and reindexing of the elastic search database everything worked.

    This was a quick and dirty solution. Is there a better way to get this to work? And also, was this unique to our installation of Azure devops, or could this happen more often?

    • Владимир Конев 0

      Hi Edwin Siebes !
      Can you provide more details how to fix it –

      search plugin with version: '6.2.4.22' is not compatible, supported search plugin version: 6.2.4.18

      Thanks!

      • edwin siebes 0

        The quick and dirty way to fix it, was to (before install) change all the references in the modules with version 6.2.4.22 to 6.2.4.18 and also rename the jar files to that version. (I’ve taken the files that come with the patched azure devops instance.
        After that run the upgrade.

    • Staffan Westin 0

      We have the exact same problem… Your solution doesn’t really appeal to me – is there a new patch coming for this perhaps? @Gloridel?

      • Gloridel MoralesMicrosoft employee 0

        Hi Staffan, for now we don’t have additional plans to address this in future patches. I will continue to post updates to this blog post with results from additional validations the team is performing.

      • Владимир Конев 0

        @Gloridel
        please advise how to fix the problem –
        search plugin with version: ‘6.2.4.22’ is not compatible, supported search plugin version: 6.2.4.18

    • Kumar Shanu (MINDTREE LIMITED)Microsoft employee 0

      Hi Edwin, As you are using two different servers for AzureDevops and Search, you have to replace Search server files with zip files of AzureDevops Server which you get after applying the patch before configuring the search.

      • Staffan Westin 0

        Could you please be a little more detailed? I don’t understand what you mean.

      • edwin siebes 0

        Hello Kumar,
        thats exactly what i did.

    • Vito A Lodese Jr 0

      Edwin,

      Thank you we hit this and your post was a big help!

  • Florian Lauger 0

    Hi Gloridel!

    We installed patch 9 for Azure DevOps Server 2020.01. and then ran “Configure-TFSSearch.ps1 -Operation update”.

    It worked that the jndilookup class was removed from the log4j binaries (see release notes “Addressed Elasticsearch vulnerability by removing the jndilookup class from log4j binaries.”).

    What didn’t work is the release note “Upgraded search plugins to use log4j core version 2.17.1.”. After installing the patch and reconfiguring the search, the old version 2.9.1 is still present. Even if you look into the patch (ZIP), you can see that the old version is still delivered.

    Is this expected or is the patch not complete/buggy?

    • Gloridel MoralesMicrosoft employee 0

      Hi Florian, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.

  • Vincent Tollu 0

    Same here, I highly doubt that a 2.17.1 version of log4J was shipped.

    First I updated ES, but saw a “C:\ES\elasticsearchv6.2\lib\log4j-core-2.9.1.jar”

    So I removed and installed again, and still this log4j-core-2.9.1.jar

    I suspected the source in C:\Program Files\Azure DevOps Server 2020\Search\zip but all is dated from today 02/03/22 so it seems to be legit

    So it seems that there’s a bug in this patch.

        Directory: C:\Program Files\Azure DevOps Server 2020\Search\zip
    
    
    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    d-----       03/02/2022     10:55                modules
    -a----       03/02/2022     10:55           7250 almsearchauthpluginv6.2.zip
    -a----       03/02/2022     10:55       11206212 almsearchicupluginv6.2.zip
    -a----       03/02/2022     10:55         204738 almsearchv6.2.zip
    -a----       26/08/2021     11:37          20545 Configure-TFSSearch.ps1
    -a----       03/02/2022     10:55       29038876 elasticsearchv6.2.zip
    -a----       04/10/2020     22:02           5225 log4j2.properties
    -a----       28/07/2021     22:02           4793 README.txt
    -a----       28/05/2020     22:02          14313 relevance.xml
    • Gloridel MoralesMicrosoft employee 0

      Hi Vincent, I have updated the blog post with a note about the plugins upgrade. We apologize for the confusion.

      • Johannes Karl Karlsson 0

        I have applied the patch and according to the process log for the patch it was successful. We still have the log4j-core-2.9.1.jar file. It seems it’s a new copy of the file, it’s a litle bit less in size than the log4j-core-2.9.1.jar on the system before the patch4.

        Our vulnerability scanner complains about this file and says JNDI Class: Found.

        Should we consider this log4j-core-2.9.1.jar file to be safe?

      • Jan Slansky 0

        I have the same question. Is 2.9.1 version used in DevOps 2020.1.1 patch 4 considered to be safe?

  • Payne, Scott 0

    I just ran the 2019 patch 13. The patch did not update elasticsearch on the remote server. We have 2 AzureDevops 2019 servers and 1 server running elasticsearch. When I go to the search/zip folder on the AzureDevOps server and run .\Configure-TFSSearch.ps1 -Operation update , I get [ERROR]: Elasticsearch not found.

    • Gloridel MoralesMicrosoft employee 0

      Hi Payne, we are in the process of validating this configuration and will update this blog post with results.

      • Payne, Scott 0

        Hi, because there is no fix for this vunlnerability, from Microsoft, any time soon, I will recommend to our team that we uninstall elastic search in it’s current configuration.

    • Gloridel MoralesMicrosoft employee 0

      Hi Payne, I have updated the blog post with a note about installations on multiple servers.

  • Muehlberger, Rainer 0

    Hello Gloridel,
    I have a question not directly about the update but about Azure DevOps Server in general.
    I think they are sitting at the right source 🙂
    Will in one of the next versions of Azure DevOps Server the XAML feature be taken out?
    Are there any thoughts on this already, though no concrete timelines?
    We are currently still using it and so I would want to know if it is being considered. It would help us prepare accordingly.

    Thank you very much.

    • Tore Østergaard Jensen (TORE) 0

      I second this question.

      • Muehlberger, Rainer 0

        Hello Jensen,
        thanks for your support. It seems, there is no clear plan available yet….

  • techatbc 0

    Hello, we currently run TFS 15.0 on prem and our vulnerability scanner has flagged this path (C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-2.4.1\lib\log4j-1.2.17.jar) for the new critical log4j vulnerability (CVE-2022-23307). Are you able to confirm if any remediation steps need to be taken for this new one as well as the original CVE-2021-44228 (or any log4j related vulnerabilities)? From what I am reading, TFS 15 is not affected by the original vulnerability but nothing on the new one. Thanks a lot!

  • Franz Haas 0

    “Patch can be installed” CheckInstall option reports that the patch is not installed! Any fixes for this?

    Checking SOFTWARE\Microsoft\TeamFoundationServer\18.0 to see if Azure DevOps Server is installed
    Found InstallPath: C:\Program Files\Azure DevOps Server 2020\
    Found InstallVersion: 18.181.31230.2
    Latest patch installed on machine is version 18.181.31609.2
    Patch 18.181.32118.5 is the same or later version as the patch installed on machine, patch can be installed.
    The Application Tier is configured.
    The Search Tier is configured.
    The Proxy Tier is not configured.
    This patch does not apply to Azure DevOps Server version 18.181.31230.2.

Feedback usabilla icon