How much is enough?<\/h3>\n
Code coverage is often used as a metric to determine whether sufficient testing has been done to “release” software. The age old question is “How much is enough?” One size does not fit all. Except in the most extreme cases, 100% is not the right answer. Think about what would be involved in 100%. You’d have to write tests to execute every single exception handler, every single branch in your code. Your code probably includes machine generated code – for example, web service SOAP proxies that you don’t even use. There are other example of “dead” code that is hard or inadvisable to remove.<\/p>\n
So, if it’s not 100%, what is it? We use a minimum bar of 70%, with the vast majority of that coming from automated tests. We only use manual testing coverage to bridge gaps in our automated testing. Would more be better? Maybe. One thing to keep in mind is that every automated test you have comes with a tax – you have to write it, run it and maintain it indefinitely (but this is the subject of a whole different blog post – so I’ll try to get to that one in the next few months).<\/p>\n
Be careful how you use code coverage. No amount of code coverage will tell you if you have a well tested app. This is because there are tons of bugs that can’t be flagged by code coverage. The problem in these cases is not that the code isn’t being covered but rather that the data variation isn’t sufficient. Buffer overruns are a very simple example of bugs that can still exist in code that has 100% code coverage. This fact really hit me most as we were building TFS and I realized how susceptible SQL is to this problem. Running every query\/stored procedure tells so little about the correctness of your SQL code. SQL depends so much on the data you have in the database and the values you query for.<\/p>\n
I’ve seen teams drive for exceptionally high code coverage numbers. Like everything else in this world, the closer you want to get to “perfect”, the harder it is. It’s much easier to go from 20%->30% than from 60%->70% and even harder to go from 80%->90%. Grinding out test after test to cover every block of code becomes an exercise of increasing effort and decreasing return (in terms of useful bugs found). At some point you are better off directing that extra effort at alternative testing approaches. For us, we’ve decided 70% is the min bar. Making that the min bar, of course means that our average is a bit higher than that.<\/p>\n
The DevDiv code coverage methodology<\/h3>\n
As I mentioned before, we focus primarily on automated testing code coverage. The reason is that we have to test on such a wide range of configurations, we must be able to rely on our automated tests to find the vast majority of problems. We break things down at the DLL\/Assembly level and our metrics are based around the percentage of component that have 70% or better code coverage. Our primary report looks like this and shows % of components and the % code coverage of that component by coverage band, trended by build:<\/p>\n
![](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image002%5B3%5D.jpg\")
<\/p>\n
We classify each componentby % code coverage bands and track the trends over time. By the time we ship Orcas, the bar should be 100% green. Our goal for Beta 1 was 75% of components over 70% code coverage. Our goal for Beta 2 is 90% of components over 70% code coverage and our goal for RTM is 100% of components over 70% code coverage.<\/p>\n
We then break that down by feature team and factor in the pass rate on each run. Obviously, it lots of tests are failing, it’s going to affect your coverage numbers.<\/p>\n
![](\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D%5B4%5D.png\")
<\/a> <\/p>\n