Some time ago, I discussed custom dialog classes. You can specify that a dialog template use your custom dialog class by putting the custom class’s name in the CLASS statement of the dialog template. A customer tried doing that but it crashes with a stack overflow.

// Dialog template

IDD_AWESOME DIALOGEX 0, 0, 170, 62
STYLE DS_SHELLFONT | DS_MODALFRAME | WS_POPUP | WS_CAPTION
CAPTION "I'm so awesome"
CLASS "MyAwesomeDialog"
FONT 8, "MS Shell Dlg", 0, 0, 0x1
BEGIN
    ICON            IDI_AWESOME,IDC_STATIC,14,14,20,20
    LTEXT           "Whee!",IDC_STATIC,42,14,114,8
    DEFPUSHBUTTON   "OK",IDOK,113,41,50,14,WS_GROUP
END

// Custom dialog class procedure
// Note: This looks ugly but that's not the point.
LRESULT CALLBACK CustomDlgProc(HWND hwnd, UINT message,
                               WPARAM wParam, LPARAM lParam)
{
    if (message == WM_CTLCOLORDLG) {
        return (LRESULT)GetSysColorBrush(COLOR_INFOBK);
    }
    return DefDlgProc(hwnd, message, wParam, lParam);
}

void Test()
{
    // Register the custom dialog class
    WNDCLASS wc{};
    GetClassInfo(nullptr, WC_DIALOG, &wc);
    wc.lpfnWndProc = CustomDlgProc;
    wc.lpszClassName = TEXT("MyAwesomeDialog");
    RegisterClass(&wc);

    // Use that custom dialog class for a dialog
    DialogBox(hInstance, MAKEINTESOURCE(IDD_AWESOME), hwndParent,
              CustomDlgProc);
}

Do you see the problem?

The problem is that the code uses the CustomDlgProc function both as a window procedure and as a dialog procedure.

When a message arrives, it goes to the window procedure. This rule applies regardless of whether you have a traditional window or a dialog. If you have a standard dialog, then the window procedure is Def­Dlg­Proc, and that function calls the dialog procedure to let it respond to the message. If the dialog procedure declines to handle the message, then the Def­Dlg­Proc function does some default dialog stuff.

Creating a custom dialog class means that you want a different window procedure for the dialog, as if you had subclassed the dialog. The custom window procedure typically does some special work, and then it passes messages to Def­Dlg­Proc when it wants normal dialog behavior.

If you use the same function as both the window procedure and the dialog procedure, then when the function (acting as a window procedure) passes a message to Def­Dlg­Proc, the Def­Dlg­Proc function will call the dialog procedure, which is also Custom­Dlg­Proc. That function doesn’t realize that it’s now being used as a dialog procedure, so it is expected to return TRUE or FALSE (depending on whether it decided to handle the message). It thinks it is still a window procedure, so it passes the message to Def­Dlg­Proc, and the loop continues until you overflow the stack.

The idea behind custom dialog classes is that you have some general behavior you want to apply to all the dialogs that use that class. For example, maybe you want them all to use different default colors, or you want them all to respond to DPI changes the same way. Instead of replicating the code in each dialog procedure, you can put it in the dialog class window procedure.

But even if you are using a custom dialog class, your dialog procedure should still be a normal dialog procedure. That dialog procedure is the code-behind for the dialog template, initializing the controls in the template, responding to clicks on the controls in the template, and so on.