{"id":112090,"date":"2026-02-26T07:00:00","date_gmt":"2026-02-26T15:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=112090"},"modified":"2026-02-26T10:04:46","modified_gmt":"2026-02-26T18:04:46","slug":"20260226-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20260226-00\/?p=112090","title":{"rendered":"Intercepting messages inside Is­Dialog­Message<\/CODE>, installing the message filter"},"content":{"rendered":"

Last time, we saw that one way to intercept the ESC<\/kbd> in the standard dialog message loop is to use your own dialog message loop<\/a>. However, you might not be able to do this, say, because the dialog procedure uses End\u00adDialog()<\/code>, and the dialog exit code is not retrievable from a custom message loop.<\/p>\n

The Is\u00adDialog\u00adMessage<\/code> includes an extensibility point that lets you hook into the message processing. You can register a message filter hook and listen for MSGF_DIALOG\u00adBOX<\/code>.<\/p>\n

Before processing a message, the Is\u00adDialog\u00adMessage<\/code> function does a Call\u00adMsg\u00adFilter<\/code> with the message that it is about to process and the filter code MSGF_DIALOG\u00adBOX<\/code>. If the filter result is nonzero (indicating that one of the hooks wanted to block default processing), then the Is\u00adDialog\u00adMessage<\/code> returns without doing anything. This lets us grab the ESC<\/kbd> from Is\u00adDialog\u00adMessage<\/code> before it turns into an IDCANCEL<\/code>.<\/p>\n

Here’s our first attempt. (There will be more than one.)<\/p>\n

HWND hdlgHook;\r\n#define DM_ESCPRESSED (WM_USER + 100)\r\n\r\nLRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam)\r\n{\r\n    if (code == MSGF_DIALOGBOX) {\r\n        auto msg = (MSG*)lParam;\r\n        if (IsDialogESC(hdlgHook, msg)) {\r\n            return SendMessage(hdlg, DM_ESCPRESSED, 0, lParam);\r\n        }\r\n    }\r\n    return CallNextHookEx(nullptr, nCode, wParam, lParam);\r\n}\r\n<\/pre>\n
Our hook procedure first checks that it’s being called by Is\u00adDialog\u00adMessage<\/code>. if so, and the message is a press of the ESC<\/kbd> key destined for our dialog box (or a control on that dialog box), then send the dialog box a DM_ESC\u00adPRESSED<\/code> message to ask it what it thinks. The dialog procedure can return TRUE<\/code> to block default processing or FALSE<\/code> to allow default processing to continue.<\/p>\n
Here is the handler in the dialog procedure itself:<\/p>\n
INT_PTR CALLBACK DialogProc(HWND hdlg, UINT message, WPARAM wParam, LPARAM lParam)\r\n{\r\n    switch (message) {\r\n    case WM_INITDIALOG:\r\n        hdlgHook = hdlg;\r\n        \u27e6 other dialog initialization as before \u27e7\r\n        \u27e6 ending with \"return (whatever)\" \u27e7\r\n\r\n    case DM_ESCPRESSED:\r\n        if (\u27e6 we want to process the ESC key ourselves \u27e7) {\r\n            \u27e6 do custom ESC key processing \u27e7\r\n            SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE);\r\n            return TRUE;\r\n        }\r\n        break;\r\n    \u27e6 handle other messages \u27e7\r\n    }\r\n    return FALSE;\r\n}\r\n<\/pre>\n
When the dialog initializes, remember its handle as the dialog for which the Dialog\u00adEsc\u00adHook\u00adProc<\/code> is operating.<\/p>\n
When the dialog is informed that the ESC<\/kbd> key was pressed, we decide whether we want to process the ESC<\/kbd> key ourselves. If so, then we do that custom processing and set up to return TRUE<\/code> from the window procedure. For dialog procedures, this is done by setting the message result to the desired window procedure result and then returning TRUE<\/code> to block default dialog box message processing and instead return the value we set (which is TRUE<\/code>) from the window procedure.<\/p>\n
Finally, we install the message hook before we create the dialog box and remove it when the dialog box dismisses.<\/p>\n
auto hook = SetWindowsHookEx(WM_MSGFILTER, DialogEscHookProc,\r\n                             nullptr, GetCurrentThreadId());\r\nauto result = DialogBox(hinst, MAKEINTRESOURCE(IDD_WHATEVER),\r\n                        hwndOwner, DialogProc);\r\nUnhookWindowsHookEx(hook);\r\n<\/pre>\n
This is the basic idea, but we see that there are a few problems.<\/p>\n
One is that we are communicating the dialog box handle through a global variable. This means that we can’t have multiple threads using this hook at the same time. Fortunately, that can be fixed by changing the variable to be thread_local<\/code>, although this does drag in the cost of thread-local variables.<\/p>\n
But even if we do that, we have a problem if two copies of this dialog box are shown by the same thread<\/i>. For example, one of the controls in the dialog might launch another copy of this dialog, but with different parameters. For example, a “View certificate” dialog might have a button called “View parent certificate”.<\/p>\n
We’ll take up these issues (and others) next time.<\/p>\n","protected":false},"excerpt":{"rendered":"
Using an Is­Dialog­Message<\/CODE> extension point.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-112090","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Using an Is­Dialog­Message<\/CODE> extension point.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/112090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=112090"}],"version-history":[{"count":1,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/112090\/revisions"}],"predecessor-version":[{"id":112091,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/112090\/revisions\/112091"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=112090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=112090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=112090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}