Given how much time we all spent at home over the past year – between the pandemic, the wildfire smoke and the murder hornets – I paid a visit last summer to my local audio dealer to shop for some new stereo equipment. I fell in love with a new McIntosh solid-state amplifier and impulsively purchased it on the spot. Or I should say, I tried to purchase it, because when the salesperson ran my credit card, we discovered that my bank had closed my account due to nonactivity.

It’s true that I hadn’t used that card for some time, but I still would have appreciated some advance notice and the opportunity to keep my account open, so I called my bank to complain. The bank informed me that they had indeed given me that advance notice and opportunity, but they had never heard back from me. It turns out the problem wasn’t that they hadn’t told me, it was that they also sent me so much other junk that I stopped paying attention to any of it. When you send me ten offers a week for discounted hotels and car rentals, I’m just not going to open every letter to closely examine it.

The same principle holds true for automated tooling bugs. If a security tool catches a critical vulnerability, but also reports 99 other findings that developers spend precious hours investigating that turn out to be false positives, developers are going to quickly learn that the tool is garbage. They’ll ignore everything that the tool reports and then miss the important issues, just like I did with my credit card mail.

As engineering system owners, this is our problem too. We can’t just point the finger at the tool developers and hope they fix things for us. We must accept end-to-end ownership of the developer’s experience with our engineering system. Our work isn’t done when bugs get filed. Our work is done when bugs get fixed. If we allow noisy tools to file false positive bugs, we’re moving away from this goal.

In 1ES, we analyzed telemetry of the automated tooling bugs that we filed to Microsoft engineering teams, and we found a Pareto effect where a tiny number of the overall static analysis rules we ran accounted for a huge proportion of “Won’t Fix, False Positive” resolutions. A great example of this is the Code Analysis rule CA2202 “Do not dispose objects multiple times.” As the rule documentation says, a properly-implemented Dispose method is safe to call multiple times, but the rule analysis isn’t sophisticated enough to determine whether a given Dispose method is properly implemented, so it just flags all occurrences where Dispose could be called multiple times. Our developers told us loud and clear – through their words and their actions – that this rule is too noisy to be valuable to them.

So, we deprecated CA2202 and other rules like it, and not only did we see an immediate drop in reported false positives, we also saw a sustained trend towards fewer and fewer false positive reports over time. Developers learned that our tooling was more trustworthy, and they gave tool findings more benefit of doubt.

If you’re looking to do this in your org, I want to emphasize that per-rule is a better granularity for your analysis than per-tool. I’ve often heard developers and security champs complain that “Tool X is super noisy”, but when we dig in, we find that it’s only some of Tool X’s rules that are noisy, and that it has some other really accurate and valuable rules that no other tool duplicates. Continuing the Code Analysis example, telemetry shows that Microsoft developers fix Code Analysis cryptography rule detections >99% of the time. It’d be foolish to deprecate the entire tool and lose the value of the crypto detections just because the multiple-dispose rule is bad.

I encourage you to identify and weed out noisy rules in the analysis tooling that you run in your organization. Just like my new amp (which I did purchase with a different card that’s in absolutely no danger of being declared inactive), the greater your signal-to-noise ratio, the sweeter your experience will be.

Want to come work with 1ES? Check out https://aka.ms/1esjobs for open positions.