Generating good error messages from code analysis tools is harder than it looks

Raymond Chen

Raymond

I was writing a tool to analyze Windows Runtime winmd files, and the tool had to generate error messages when it discovered something that violated our team’s internal rules.

It turns out that generating good error messages from code analysis tools is hard. My tool detected, for example, that the name of a method parameter was improperly changed. (Method parameter names are used by some language projections, so changing the name of the parameter is a breaking change.)

It was hard enough detecting that a change occurred that violated our team’s internal rules. Now came the problem of reporting it.

My original code returned a Boolean value that said whether everything was okay, but it’s obviously a bad idea for the tool to simply say “Sorry, you made a mistake. Now go fix it.” I had to modify my code to report enough information in order to diagnose the problem and fix it. This means that instead of just “Sorry, you made a mistake,” the error message was something like “Disallowed change of parameter name from ‘index’ to ‘i’ in parameter 1 of method GetAt of interface IVector (with T = string) implemented by class MyVector in file Sample.winmd, compared to Baseline.winmd.”

I developed an appreciation for how hard it is to generate good error messages, especially if the thing you’re reporting is not something that a human being wrote directly, which means that you can’t use line numbers as reference points.

 

Raymond Chen
Raymond Chen

Follow Raymond   

7 Comments
Avatar
Alexey Badalov 2019-07-04 08:32:46
I discovered this when they asked us to write a Pascal compiler in school. Error reporting is a huge and complicated area.
Avatar
Neil Rashbrook 2019-07-05 03:22:23
I followed the link, but I didn't spot any renamed parameters, let alone how the rename would break anything...
Avatar
David Walker 2019-07-06 15:08:44
I'm extremely glad that my "connect" item, from 2006, about a poor error message in SQL is finally getting fixed.  (When inserting data into a SQL table, the old error message said "String or binary data would be truncated" but did not say which column.  The new error message gives the name of the column and some of the data.  That's very helpful when there may be scores of columns in the table.)    Yes, writing error messages is hard.    Sometimes it takes more than a decade to improve an error message...
Avatar
smf esemef 2019-07-08 02:36:42
If you want humans to understand it, then you should generate the same kind of words that developers think when they review code. Just pick a handful of random explitives and insults and you'll be fine. Bonus points if you can generate an image macro.