One of my colleagues was debugging a program and discovered that a call to SetÂFocus returned NULL, which the documentation calls out as indicating an error condition. However, a follow-up call to GetÂLastÂError() returned 0, which means “Everything is just fine.”
After much debugging, they figured it out: There was a WH_CBT hook that was intercepting the SetÂFocus call and rejecting the focus change by returning TRUE.
“It would have been nice to have received a useful error code like ERROR_YOU_JUST_GOT_SCREWED_BY_A_HOOK.”
You don’t get a useful error because the window manager doesn’t know that the hook is screwing with you. You called SetÂFocus. The window hook said, “Nope, don’t change focus. It’s all good. No worries.” The window manager says, “Okay, well, then I guess it’s all taken care of.”
Hooks let you modify or even replace certain parts of the window manager. If you do that, then it’s on you to do so in a manner that will not confuse the rest of the system. If your hook wants to make SetÂFocus fail, and not set an error code, we’ll that’s your decision. The system is not going to call SetÂLastÂError(ERROR_YOU_JUST_GOT_SCREWED_BY_A_HOOK) because that might overwrite an error code set by the hook.
In this specific example, the point of the WH_CBT hook is to assist with computer-based training: The program installs a CBT hook, which can then do things like prevent the program from changing focus so that the window containing the training materials retains focus. The underlying assumption is that a CBT hook is going to mess around only with windows that it is already in cahoots with.
“Oh, this is my print dialog. I’m going to prevent it from taking focus so that my instructions on how to use the printer stays on screen with focus. I’m also going to make changes to my print dialog function so it doesn’t freak out when it fails to get focus.”
Whatever program installed this CBT hook didn’t limit their meddling to windows they already controlled. This means that their actions sowed confusion among other windows that weren’t part of their little game.
I suspect no actual computer-based training was going on at all. The CBT hook was being used not for its stated purpose of computer-based training, but rather because it provided a way to alter the behavior of the window manager in very fundamental ways, and being able to make those alterations fit into the program’s world view somehow.
Somebody who installs a hook can alter the behavior of the system, and it’s important that they do it right, so that their changes still maintain the contracts promised by the system. One of those being that when SetÂFocus fails, it tells you why.
Related reading: The case of the file that won’t copy because of an Invalid Handle error message.
The only documentation I could find for the
WH_CBThook doesn’t say anything about having to set the last error if you’re rejecting the focus change, so whose job is it to set the last error in such a case?The problem with the documentation is that it is written under the assumption that you are using it for the right thing. This means that everyone is in on the joke.
If you aren't then it would be the hook's responsibility. The way that hooks work is that when the messages are sent, it first calls any hooks installed and if the hook doesn't block the action, it then does the work and post the messages. The only bits of code that are capable of calling SetLastError would be the hook procedure and the Windows API that called into the...
Typo: “we’ll that’s your decision”
Was the hook in the same process? Could you describe the diagnostic steps? What tools were used?