Adventures in application compatibility: Following a detour off a cliff

The application compatibility team reported that a program was crashing in its installer. Debugging the installer revealed that it actually triggered three exceptions.

The first exception was due to passing an invalid pointer to the lstrlen function. However, this didn’t crash the installer because the lstrlen function contains an application compatibility mitigation: If you pass a pointer to invalid memory, it handles the access violation exception by reporting that the string length is zero.

It did this twice, so that is the second exception as well.

The third exception is the one that crashed the installer: A system function wanted to report a runtime issue. It did this by checking if a debugger was present, and if so, it printed a diagnostic message to the debugger so that the developer could see that there was a problem and get a chance to investigate it.

The problem is that the call to Is­Debugger­Present was crashing.

And it was crashing because the installer used a DLL which detours the Is­Debugger­Present function when it is loaded, presumably to make it harder to reverse-engineer (though since the source code is freely available, it’s not sure why they are so concerned about it).

But notice that the DLL does not un-detour the function when it is unloaded.¹

Therefore, if you load the DLL and then unload it, you leave the Is­Debugger­Present function detoured to a function that no longer exists. And the next person to call the Is­Debugger­Present function will crash.

(Also curious is that the DLL is licensed under the GPL, yet the product that uses it makes no mention of the GPL in its license agreement.)

The fix was to remove the debugging message for the particular case that this program’s installer was tripping over.

¹ Not that un-detouring is a perfect solution either, because it runs into the What if two programs did this? problem: Suppose another DLLs also detoured the Is­Debugger­Present function. When this DLL unloads, it restores the original function, which also removes the other DLL’s detour. And then when the other DLL unloads, it restores what it thought was the original function, but which was actually the detour installed by this DLL. As a result, this DLL’s detour gets reinstalled.