There are a few different built-in ways to close a dialog box in the classic Windows dialog manager. Let’s run them down.

First, there’s hitting the ESC key.

The ESC key, as with all keyboard navigation, is handled by the Is­Dialog­Message function. Assuming that the dialog control with focus did not use the WM_GET­DLG­CODE message to override default keyboard handling, the Is­Dialog­Message function converts the ESC to a simulated button click of whatever dialog control has the ID IDCANCEL. Specifically, the message is WM_COMMAND, the notification code is BN_CLICKED, the control ID is IDCANCEL, and the window handle is the handle to whatever dialog control has the ID IDCANCEL (or nullptr if there is no such control).

Exception: If there is a control whose ID is IDCANCEL, and that control is disabled, then the Is­Dialog­Message function merely beeps and otherwise ignores the ESC key.

Okay, what about the Close button in the title bar, the one that looks like an ×?

The Close button in the title bar, double-clicking the dialog box icon (if there is one), selecting Close from the system menu, and pressing Alt+F4 all behave the same way: They generate a WM_SYSCOMMAND message whose wParam & 0xFFF0 is SC_CLOSE. The default window procedure turns this into a WM_CLOSE message. The default dialog procedure responds to the WM_CLOSE in basically the same way that Is­Dialog­Message does: It generates a simulated button click of whatever dialog control has the ID IDCANCEL. Again, this is done by converting it to the WM_COMMAND message, with a notification code of BN_CLICKED, a control ID of IDCANCEL, and the handle to whatever dialog control has the ID IDCANCEL (or nullptr if there is no such control). It also has the same exception: If there is a control whose ID is IDCANCEL, and that control is disabled, then the default dialog procedure just beeps and otherwise ignores the message.

Now that we understand what happens, next time we can look at ways of customizing the behavior.

Bonus chatter: You can see from this that the dialog manager is wired to treat a control with the ID IDCANCEL as if it were a Cancel button, so if you have a Cancel button, give it the ID IDCANCEL. Conversely, if you have a control whose ID is IDCANCEL, it had better be a button if you know what’s good for you.