We started by exploring the signals the dialog manager uses for dismissing a dialog. Now we can use that information to customize the dismiss behavior.

Let’s start with a diagram, because people like diagrams.

We noted at the end of that article that if you have a button whose ID is IDCANCEL, the dialog manager will defer to that button to decide whether to allow ESC key, the Close button, or other nonclient affordances to convert the corresponding action to a simulated click on that button. So the obvious way to take advantage of this is to put a Cancel button on your dialog box, and disable it when you don’t want the user to dismiss the dialog box with the ESC key, the Close button, or other standard affordances.¹

Notice that everything in the diagram funnels into WM_COMMAND/IDCANCEL, even if you don’t actually have a control whose ID is IDCANCEL. If you add a handler to your dialog procedure for WM_COMMAND, then all of the actions will come through that handler, and you can customize the behavior at that point. You could call EndDialog to close the dialog or just return without doing anything to keep the dialog open.

Now, if you have no intention of closing the dialog in response to the Close button or the system menu Close command, then you probably shouldn’t leave them enabled. You can gray those out by doing

    EnableMenuItem(GetSystemMenu(hDlg, FALSE), SC_CLOSE,
                   MF_BYCOMMAND | MF_DISABLED | MF_GRAYED);

Okay, but what if you want to treat the Close button differently?

From the diagram, we see that the Close button goes through a WM_CLOSE phase, so you can handle the WM_CLOSE message in your dialog procedure to do whatever custom Close button behavior you want. If you return TRUE from the dialog procedure, then that will mark the message as handled, and further processing will stop. if you return FALSE, then the Def­Dlg­Proc will turn the WM_CLOSE into the WM_COMMAND message as before.

If you want to treat the ESC key differently from a click on the Cancel button, you’ll have to intercept it on the Is­Dialog­Message path. We’ll look at that next time.

¹ Now, you might notice that there is no requirement that the IDCANCEL button be in the tab order, or that it even be visible. The dialog manager merely checks whether it is enabled. Therefore, you might be tempted to control these actions by disabling your (invisible, non-tabbable) IDCANCEL button. This is sneaky, but it will probably confuse assistive technology tools, and anybody else who inspects the window hierarchy, and besides, it’s more complicated than the other alternatives presented here.