Last time, we looked at recognizing that the user clicked the Close button (or equivalent nonclient gestures) on a dialog. The other system-provided pathway to dismissing a dialog is pressing ESC, and we saw in our flow diagram that this is done by the Is­Dialog­Message function.

If your dialog box doesn’t have an IDCANCEL button, then you can detect that the user hit ESC by simply recognizing that they didn’t click the Close button. If you shut off the WM_CLOSE pathway, then the only other source of IDCANCEL is the ESC key.

Now, you might be concerned that additional pathways for system-provided dialog dismissal may be added later. (Who knows, maybe a new touch gesture will be invented.) But you can’t predict what pathway that future system-provide dismissal will take, so you have nothing to code to. All you can do is cover the pathways that you know and hope that any future dismissal mechanisms will follow one of them.

We saw from our diagram that the ESC pathway consists of the Is­Dialog­Message function processing a WM_KEY­DOWN for the ESC key and turning it into a WM_COMMAND button click for IDCANCEL. By the time it is converted into a fake button click message, it’s too late to know what the source was, so we’ll have to do the work before then.

One way to do this is to recognize the ESC key before calling IsDialogMessage. If your dialog was created as a modeless dialog, then you already have a custom dialog message loop. And if it was created as a modal dialog, you can convert it to a modeless one with a dialog message loop. Once that’s done, you can treat the ESC key as if it were custom navigation.

Your first try might go like this:

while (⟦ dialog still active ⟧ &&
       GetMessage(&msg, NULL, 0, 0, 0)) {
 if (msg.message == WM_KEYDOWN &&
     msg.wParam == VK_ESCAPE) {
  ⟦ do custom ESC key handling ⟧
 } else if (!IsDialogMessage(hdlg, &msg)) {
  TranslateMessage(&msg);
  DispatchMessage(&msg);
 }
}

However, this fails to honor controls that declare DLGC_WANT­ALL­KEYS or DLGC_WANT­MESSAGE. We’ll have to check with the focus control to see if it wants to use the ESC key for its own purposes. While we’re at it, we’ll also ensure that we are stealing ESC keys only from our own dialog. The code is getting complex enough that we’ll break it out into a helper function.

bool IsDialogESC(HDLG hdlg, MSG const* msg)
{
    if (msg->message != WM_KEYDOWN ||
        msg->wParam != VK_ESCAPE) {
        return false;
    }

    if (msg->hwnd != hdlg &&
        !IsChild(hdlg, msg->hwnd)) {
        return false;
    }

    auto code = SendMessage(msg->hwnd, WM_GETDLGCODE,
                            msg->wParam, (LPARAM)msg);
    if (code & (DLGC_WANTALLKEYS | DLGC_WANTMESSAGE)) {
        return false;
    }

    return true;
}

In order for this to be a potential dialog-dismissing ESC, the message must be a WM_KEY­DOWN of VK_ESCAPE, and the message must target the dialog or a child of the dialog. And the target window also must decline to handle the message itself.

I carefully ordered the tests so that we can early-out as quickly as possible. Checking for the ESC key can be done by inspecting the message. Checking that the target window is acceptable is a little more work, but not too bad. Checking whether the target window wants to handle the message is the most expensive test, so we do that last.

Now we can incorporate this helper function into our custom message loop.

while (⟦ dialog still active ⟧ &&
       GetMessage(&msg, NULL, 0, 0, 0)) {
 if (IsDialogESC(hdlg, &msg)) {
  ⟦ do custom ESC key handling ⟧
 } else if (!IsDialogMessage(hdlg, &msg)) {
  TranslateMessage(&msg);
  DispatchMessage(&msg);
 }
}

This all sounds great, but you might not be able to make this change. For example, maybe the dialog box’s dialog procedure uses End­Dialog() to dismiss the dialog. The fact that it has been called is not exposed by the dialog box infrastructure. Or maybe you don’t control the code that calls Dialog­Box in the first place, so you can’t make them switch to a modeless dialog box with a custom dialog loop.

We’ll study this problem next time.