Generating good error messages from code analysis tools is harder than it looks
I was writing a tool to analyze Windows Runtime winmd files, and the tool had to generate error messages when it discovered something that violated our team’s internal rules.
It turns out that generating good error messages from code analysis tools is hard. My tool detected, for example, that the name of a method parameter was improperly changed. (Method parameter names are used by some language projections, so changing the name of the parameter is a breaking change.)
It was hard enough detecting that a change occurred that violated our team’s internal rules. Now came the problem of reporting it.
My original code returned a Boolean value that said whether everything was okay, but it’s obviously a bad idea for the tool to simply say “Sorry, you made a mistake. Now go fix it.” I had to modify my code to report enough information in order to diagnose the problem and fix it. This means that instead of just “Sorry, you made a mistake,” the error message was something like “Disallowed change of parameter name from ‘index’ to ‘i’ in parameter 1 of method GetAt of interface IVector (with T = string) implemented by class MyVector in file Sample.winmd, compared to Baseline.winmd.”
I developed an appreciation for how hard it is to generate good error messages, especially if the thing you’re reporting is not something that a human being wrote directly, which means that you can’t use line numbers as reference points.
Light
Dark
7 comments
I discovered this when they asked us to write a Pascal compiler in school. Error reporting is a huge and complicated area.
I followed the link, but I didn’t spot any renamed parameters, let alone how the rename would break anything…
I could see a problem where you’re naming your arguments, so if I have some code:
myService.Execute(actionName: “Fetch”)
It’s going to depend on the declaration being ‘Execute(string actionName)’ and would break if the owner of Execute changed the decl to ‘Execute(string action)’.
“AllJoyn: Refresh generated code to fix build breaks.” Somebody renamed a parameter, and that broke the AllJoyn samples.
I’m extremely glad that my “connect” item, from 2006, about a poor error message in SQL is finally getting fixed. (When inserting data into a SQL table, the old error message said “String or binary data would be truncated” but did not say which column. The new error message gives the name of the column and some of the data. That’s very helpful when there may be scores of columns in the table.)
Yes, writing error messages is hard.
Sometimes it takes more than a decade to improve an error message…
It’s opt-in via a trace flag and I wonder if it will never be on by default. There is an argument to be made that displaying that information to an end user is a potential security problem. One recommendation I read was to keep it off, then turn it on when you need it for troubleshooting purposes.
I am glad your “connect” item was finally addressed. It was a frustrating error to try to diagnose.
If you want humans to understand it, then you should generate the same kind of words that developers think when they review code.
Just pick a handful of random explitives and insults and you’ll be fine. Bonus points if you can generate an image macro.