Last time, we used a MSGF_DIALOG­BOX message filter to hook into the Is­Dialog­Message so that we had the option to grab the ESC before it gets turned into an IDCANCEL. There are some problems with our initial foray.

One is the problem of recursive dialogs. If the first dialog shows another copy of itself (for example, a certificate dialog showing a dialog for its parent certificate), then the thread-local variable gets overwritten, and the first dialog’s information is lost.

We could solve that by having each dialog remember the original value and restore it when the dialog dismisses. Alternatively, we could maintain an explicit stack of dialogs, pushing when a new dialog is created and popping when it is destroyed.

However, this fails to handle the case where the dialog is modeless. In that case, the two dialogs could be running concurrently rather than recursively. Instead of a stack, we really need a per-thread set of active dialogs.

Another thing to worry about is that if this code is put into a static library, and two components in the same thread both use that static library, then you have to be careful that the two copies of the library don’t conflict with each other.

I came up with this initial idea:

#define DIALOG_WANTS_ESC_PROP TEXT("DialogWantsEsc")

LRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (nCode == MSGF_DIALOGBOX) {
        auto msg = (MSG*)lParam;
        if (msg->message == WM_KEYDOWN &&
            msg->wParam == VK_ESCAPE) {
            auto hdlg = GetAncestor(msg->hwnd, GA_ROOT);
            auto customMessage = PtrToUint(GetProp(hdlg,
                                           DIALOG_WANTS_ESC_PROP));
            if (customMessage &&
                !(SendMessage(msg->hwnd, WM_GETDLGCODE,
                             msg->wParam, lParam) &
                         (DLGC_WANTALLKEYS | DLGC_WANTMESSAGE))) {
                return SendMessage(hdlg, customMessage, 0, lParam);
            }
        }
    }
    return CallNextHookEx(nullptr, nCode, wParam, lParam);
}

The idea here is that instead of having to manage a table of per-thread registrations, we just let dialogs self-register by setting the DIALOG_WANTS_ESC_PROP property to the message number they want to receive when the user presses ESC.

If there are two copies of this hook installed, then the Dialog­Esc­Hook­Proc is called twice. The first one sends the custom message and gets the dialog’s response, and returns it; it never passes the message down the hook chain. Therefore, the second and subsequent hooks never get to run, so we don’t have a problem of the custom message getting sent multiple times for the same call to Is­Dialog­Message.

This design has the advantage that multiple DLLs using this pattern can coexist because the first hook (whichever it is) does all the work for everybody.

An alternate, more complex, design would pass the call down the chain if the dialog box declined to handle the ESC key, in case some other hook wanted to do something special. The catch is that if there are multiple copies of this hook installed, each one will send the custom message to the dialog, which would be bad if the handler for the custom message had side effects like showing a confirmation dialog.

So we can add the rule that the custom message must be safe to call multiple times if it returns FALSE. This means that if it wants to display a confirmation dialog, it should always return TRUE even if the user cancels.

LRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (code == MSGF_DIALOGBOX) {
        auto msg = (MSG*)lParam;
        if (msg->message == WM_KEYDOWN &&
            msg->wParam == VK_ESCAPE) {
            auto hdlg = GetAncestor(msg->hwnd, GA_ROOT);
            auto customMessage = PtrToUInt(GetProp(hdlg,
                                           DIALOG_WANTS_ESC_PROP));
            if (customMessage &&
                !(SendMessage(msg->hwnd, WM_GETDLGCODE,
                             msg->wParam, msg) &
                         (DLGC_WANTALLKEYS | DLGC_WANTMESSAGE)) &&
                 SendMessage(hdlg, customMessage, 0, lParam)) {
                 return TRUE;                                  
            }
        }
    }
    return CallNextHookEx(nullptr, nCode, wParam, lParam);
}

Or we can have the first hook leave a note for the other hooks that the message has already been handled and that they shouldn’t try to handle it again.

#define DIALOG_WANTS_ESC_PROP TEXT("DialogWantsEsc")
#define CURRENT_MESSAGE_PROP TEXT("DialogWantsEscCurrentMessage")

LRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam)
{
    if (code == MSGF_DIALOGBOX) {
        auto msg = (MSG*)lParam;
        if (msg->message == WM_KEYDOWN &&
            msg->wParam == VK_ESCAPE) {
            auto hdlg = GetAncestor(msg->hwnd, GA_ROOT);
            auto customMessage = PtrToUInt(GetProp(hdlg,
                                           DIALOG_WANTS_ESC_PROP));
            if (customMessage) {
                auto previous = GetProp(hdlg, CURRENT_MESSAGE_PROP);
                if (previous != msg &&                              
                    !(SendMessage(msg->hwnd, WM_GETDLGCODE,
                                 msg->wParam, msg) &
                             (DLGC_WANTALLKEYS | DLGC_WANTMESSAGE))) {
                    return SendMessage(hdlg, customMessage, 0, lParam);
                }
                SetProp(hdlg, CURRENT_MESSAGE_PROP, msg);                     
                auto result = CallNextHookEx(nullptr, nCode, wParam, lParam); 
                SetProp(hdlg, CURRENT_MESSAGE_PROP, previous);                
                return result;                                                
            }
        }
    }
    return CallNextHookEx(nullptr, nCode, wParam, lParam);
}

The first hook will send the message to the dialog. and if the dialog declines to handle it, it passes the messages to the other hooks, but setes the “current message” property to the message that was already handled, so that other hooks won’t try to handle it again.

The last part of the puzzle is installing the hook. Since we are assuming that we cannot alter the dialog loop, the hook has to be installed by the dialog itself.

Let’s assume that this dialog box already allocates other dialog state, so we can add the hook handle to the state structure.

struct DIALOGSTATE
{
    wil::unique_hhook escapeHook;
    ⟦ other stuff ⟧
};

// each dialog can choose its own custom message
#define DM_ESCPRESSED (WM_USER+1000)

INT_PTR CALLBACK DialogProc(HWND hdlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message) {
    case WM_INITDIALOG:
        {
            DIALOGSTATE* state = new(std:nothrow) DIALOGSTATE();
            if (!state) { EndDialog(hdlg, -1); return FALSE; }
            SetWindowLongPtr(hdlg, DWLP_USER, (LONG_PTR)state);
            state->escapeHook.reset(SetWindowsHookEx(WM_MSGFILTER,     
                             DialogEscHookProc,                        
                             nullptr, GetCurrentThreadId()));          
            SetProp(hdlg, DIALOG_WANTS_ESC_PROP,                       
                    IntToPtr(DM_ESCPRESSED));                          
            ⟦ other dialog initialization as before ⟧
            ⟦ ending with "return (whatever)" ⟧
        }

    case DM_ESCPRESSED:
        if (⟦ we want to process the ESC key ourselves ⟧) {
            ⟦ do custom ESC key processing ⟧
            SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);
            return TRUE;
        }
        break;

    case WM_DESTROY:
        {
            auto state = (DLGSTATE*)GetWindowLongPtr(hdlg, DWLP_USER);
            delete state;
        }
        break;

    ⟦ handle other messages ⟧
    }
    return FALSE;
}

The dialog installs the hook when it is created and removes it when it is destroyed. The hook has become an implementation detail of the dialog.

Now, I don’t recommend doing all this. Better is to just treat with the ESC like any other press of the (possibly imaginary) Cancel button. One of the few scenarios I can think of where this could be useful is if you want to display an extra confimation for the Close button (since its meaning is potentially ambiguous). This is still nonstandard, but at least it’s not too nonstandard. And for that, you can just intercept WM_CLOSE instead of trying to intercept the ESC. Intercepting the ESC was really just an excuse to show off message filters, which tend to be unappreciated.