In software development, testing is an integral part of the development process. Tests are used to verify that software behaves as expected, and they ensure that changes made to the code do not break existing functionality. However, there is a problem that often arises when running automated tests: Flaky tests.
Flaky tests are tests that produce different results each time they are run, even when the code being tested has not changed. This can happen for a variety of reasons, such as race conditions, non-deterministic behaviour, or environmental factors. Flaky tests can be frustrating and time-consuming to deal with, and they can undermine confidence in the testing process.
The goal of bringing flaky test management in-product is to reduce developer pain cause by flaky tests and cater to the whole workflow. Azure DevOps provides 2 types of flaky test detection settings: System detection and Custom detection. Read more about them in our official documentation.
Issue with previous experience:
Users can mark or unmark a test as flaky based on analysis or context and this tagging is stored in our database. Users can also change the setting from System detection to Custom detection and vice-versa. However, all the tests that had been previously marked as flaky under system detection automatically get marked as flaky under custom detection as well. This can lead to issues as incorrect test cases have been marked as flaky.
New experience:
Going forward, a setting change in Flaky test detection removes the list of test cases that had been previously marked as flaky. Hence, if a user changes Flaky test management setting, they will have to mark all test cases as flaky again. A confirmation alert has also been added to inform users of the impact of the setting change.
Conclusion:
Azure DevOps now provides the ability to delete past history of flaky test cases on changing the settings of it’s Flaky Test management feature so that developers can improve the reliability of their automated tests
Quick demo of feature:
Step 1: Goto Project Settings -> Test Management. Step 2: Make sure ‘Flaky Test Detection’ is turned on. Step 3: Select between ‘System Detection’ or ‘Custom Detection’. Step 4: The confirmation dialog alerts users regarding deletion on existing tests that have been marked as flaky.
Hi Ravi
Improvements to the flaky test detection is always welcome! We are using Azure DevOps Server so we get these improvements a little delayed.
I am wondering if you are also looking into flakiness in the flaky detection itself e.g.:
https://developercommunity.visualstudio.com/t/VSTest-rerun-logic-have-issues-with-Data/10292590
An unrelated 'funny' finding we have done is a test that consequently failed during it first attempt because of some bad setup logic. So with rerun configured on our pipeline it never gave a failure, showing...
Hi Tore,
Thank you for your feedback and raising concerns around our Flaky Test detection experience. The ticket provided by you is in ‘Triaged’ state currently and I have shared it again with our concerned engineering team. Any updates to the request will be made on the developer community ticket itself. We hope to come back with an update soon.
Regards
Ravi