Test execution, especially in the automated testing workflow, generates considerable amount of data. To keep your test system performant and responsive, its recommended to periodically clean up test results that are no longer relevant. Below are sources from which test results are generated in TFS, and the current features (as of TFS 2015 RTM or prior versions) available to clean-up test results:

• Automated test results from XAML Builds: Automated testing in the build workflow generates significant amount of test result data. For XAML builds, though there exists an option of deleting test results when build is deleted, the option is disabled by default. This is because the XAML build system, by default, keeps only 10 latest builds. 10 builds can happen very quickly and test results may be cleaned up before they are analyzed. As a result, XAML build definitions pile up lot of orphaned test result data in the system. Such orphaned test results today have to be cleaned up using tools like the Test Attachment Clean Power Tool. In-product support to clean up such orphaned test results is not available today.  
• Automated test results from builds in the new Build System (Build vNext):  Build vNext, which shipped with TFS 2015 RTM, doesn’t support clean up of test result data, yet.
• Manual test results: Manual test results from Test Plans, not associated with builds, are kept in the system permanently and need to be cleaned up using the Test Attachment Clean Power Tool

From the above points, it is apparent that in-product support is required to cleanup test results that are old and no longer required. Also, the Test Attachment Cleaner Power Tool mentioned above cleans up only test attachments, and does not delete test run and test result entries. With TFS 2015 Update 1, we are enabling the capability to cleanup all test result data based on age. We are also integrating test result retention with the new Build system (Build vNext). 

New Test Result Retention features with TFS 2015 Update 1

1. Team project level test retention policy to clean up test result data by age

This feature, which is configurable at team project level, supports the capability to clean up all test result data in the system including test runs, test results, and test attachments that are older than a specified number of days. Separate limits can be specified for automated and manual test results, offering the flexibility to retain manual test results longer than automated test results, if required. 

 2. Build vNext level test retention policy to delete test results when builds are deleted

With TFS 2015 Update 1, you can configure the new build system to clean up test results when builds are deleted. This setting is enabled by default for new build definitions created after upgrading to TFS 2015 Update 1. Unlike XAML builds, with which it was difficult to forecast the number of days for which builds and associated test results would be available (since retention was based on build count), the retention policy of the new build system ensures certainty by letting users specify number of days to keep builds and associated test results.

Scenario walk-through

To better understand how to use the above features to configure test retention, lets walk-through a scenario. The scenario will address these set of requirements:

1. Have an aggressive test retention policy for branches that generate lot of builds and associated test result data. Delete test results associated with builds from branches like master and feature branches, when builds from such branches are deleted. 
2. Keep test results from release branch (used for shipping to production) for about 2 months, even if builds from release branch are deleted in say 20 days.
3. Keep manual test results permanently.  

 Requirement 1: A lot of builds are generated on branches like master and feature branches created by users, resulting in lot of test result data. The administrator would like to keep such builds and associated test results for a relatively short duration of, say,  5 days.

• Solution: This requirement can be serviced by having a filter on a master branch as shown in the below screenshot. Notice that ‘Delete test results’ is checked for the rule on master branch. 

Requirement 2 – part A – Keep test results from release branch even after release builds are deleted: Builds that get released and deployed to production must be retained for 20 days. If some bugs are reported from customers, engineers may want to analyze the testing done against the deployed build. As such, test results for release builds must be retained even after such builds are deleted. They should be available for around 2 months.

• Solution: This requirement can be serviced by having a filter on a master branch as shown below. As you may notice, the ‘Delete test results’ is unchecked so that test results are retained even after build is deleted. Unchecking ‘Delete build record’ ensures that there is an entry point to test results from the build summary, but bulky build artifacts such as build drop and logs are deleted. Note that branch level filters can be used if you use GIT for version control. 

**Requirement 2 – part B – Clean up left over results from ‘part A’ after 2 months: **Test results may be left over after builds are deleted, resulting in orphaned test results. As explained in the introduction section above, XAML build definitions are more likely to generate orphaned test result data since the test result deletion is disabled by default in XAML build definitions. 

• Solution: The configuration to clean up test results after 2 months can be configured at team project level. Note that this configuration is applicable all automated test results in the team project. including orphaned test results from other build definitions or automated test runs published using REST APIs that are not associated with a build. 

Requirement 3: Finally, manual test results must never be deleted. There is lots of important commentary added by the engineer while running manual tests. Also, from a compliance standpoint it may be required to justify the manual testing done at some future date.

• Solution: This configuration is simple – set the manual retention age to ‘Never delete’ at the team project level.

Behavior for existing and new team projects

For team projects that existed before upgrading to TFS 2015 Update 1, the team project level age limits for test retention are set to ‘Never delete’ for both manual and automated test results. Make sure to work with your TFS administrators to have this configured so that old test data is cleaned up from your TFS server, freeing up valuable storage space. For team projects that are created after migrating to TFS 2015 Update 1, the default value for the team project level age limits for rest retention are set to 365 days for both manual and automated test results. 

Closing

To summarize, with TFS 2015 Update 1, we have added test result data cleanup capabilities that can delete test results when builds are deleted, or at a later point of time based on the age of test results. With these capabilities, you will no longer need to use tools like the Test Attachment Cleaner. 

-Manoj Bableshwar (manoj.bableshwar at microsoft.com)