{"id":10723,"date":"2011-05-06T07:00:00","date_gmt":"2011-05-06T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2011\/05\/06\/a-function-pointer-cast-is-a-bug-waiting-to-happen\/"},"modified":"2011-05-06T07:00:00","modified_gmt":"2011-05-06T07:00:00","slug":"a-function-pointer-cast-is-a-bug-waiting-to-happen","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20110506-00\/?p=10723","title":{"rendered":"A function pointer cast is a bug waiting to happen"},"content":{"rendered":"<p>\nA customer reported an application compatibility bug in Windows.\n<\/p>\n<blockquote CLASS=\"q\">\n<p>\nWe have some code that manages a Win32 button control.\nDuring button creation, we subclass the window by calling\n<code>Set&shy;Window&shy;Subclass<\/code>.\nOn the previous version of Windows, the subclass procedure\nreceives the following messages, in order:\n<\/p>\n<ul>\n<li><code>WM_WINDOWPOSCHANGING<\/code>\n<li><code>WM_NCCALCSIZE<\/code>\n<li><code>WM_WINDOWPOSCHANGED<\/code>\n<\/ul>\n<p>\nWe do not handle any of these messages and pass them through\nto <code>Def&shy;Subclass&shy;Proc<\/code>.\nOn the latest version of Windows,\nwe get only the first two messages,\nand <code>comctl32<\/code> crashes while it&#8217;s handling\nthe third message before it gets a chance to call us.\nIt looks like it&#8217;s reading from invalid memory.\n<\/p>\n<p>\nThe callback function goes like this:\n<\/p>\n<pre>\nLRESULT ButtonSubclassProc(\n    HWND hwnd,\n    UINT uMsg,\n    WPARAM wParam,\n    LPARAM lParam,\n    UINT_PTR idSubclass,\n    DWORD_PTR dwRefData);\n<\/pre>\n<p>\nWe install the subclass function like this:\n<\/p>\n<pre>\nSetWindowSubclass(\n    hwndButton,\n    reinterpret_cast&lt;SUBCLASSPROC&gt;(ButtonSubclassProc),\n    id,\n    reinterpret_cast&lt;DWORD_PTR&gt;(pInfo));\n<\/pre>\n<p>\nWe found that if we changed the callback function declaration to\n<\/p>\n<pre>\nLRESULT <font COLOR=\"blue\">CALLBACK<\/font> ButtonSubclassProc(\n    HWND hwnd,\n    UINT uMsg,\n    WPARAM wParam,\n    LPARAM lParam,\n    UINT_PTR idSubclass,\n    DWORD_PTR dwRefData);\n<\/pre>\n<p>\nand install the subclass function like this:\n<\/p>\n<pre>\nSetWindowSubclass(\n    hwndButton,\n    <font COLOR=\"blue\">ButtonSubclassProc<\/font>,\n    id,\n    reinterpret_cast&lt;DWORD_PTR&gt;(pInfo));\n<\/pre>\n<p>\nthen the problem goes away.\nIt looks like the new version of Windows introduced\na compatibility bug; the old code works fine on all previous\nversions of Windows.\n<\/p>\n<\/blockquote>\n<p>\nActually, you had the problem on earlier versions of Windows, too.\nYou were just lucky that the bug wasn&#8217;t a crashing bug.\nBut now it is.\n<\/p>\n<p>\nThis is a classic case of\n<a HREF=\"http:\/\/blogs.msdn.com\/oldnewthing\/archive\/2004\/01\/15\/58973.aspx\">\nmismatching the calling convention<\/a>.\nThe <code>SUB&shy;CLASS&shy;PROC<\/code> function is declared as requiring\nthe <code>CALLBACK<\/code> calling convention (which on x86\nmaps to <code>__stdcall<\/code>),\nbut the code declared it without any calling convention at all,\nand the ambient calling convention was <code>__cdecl<\/code>.\nWhen they went to compile the code, they got a compiler error\nthat said something like this:\n<\/p>\n<p>\n<code>\nerror C2664: 'SetWindowSubclass' : cannot convert parameter 2 from\n 'LRESULT (__cdecl *)(HWND,UINT,WPARAM,LPARAM,UINT_PTR,DWORD_PTR)'\n to 'SUBCLASSPROC'\n<\/code>\n<\/p>\n<p>\n&#8220;Since the compiler was unable to convert the parameter,\nlet&#8217;s give it some help and stick a cast in front.\n<a HREF=\"http:\/\/blogs.msdn.com\/oldnewthing\/archive\/2009\/10\/23\/9911891.aspx#9912158\">\nThere, that shut up the compiler<\/a>.\nThose compiler guys are so stupid.\nThey can&#8217;t even figure out how to convert one function pointer\nto another.\nI bet they need help wiping their butts when they go to the bathroom.&#8221;\n<\/p>\n<p>\nAnd there you go, you inserted a cast to shut up the compiler\nand masked a bug instead of fixing it.\n<\/p>\n<p>\nThe only thing you can do with a function pointer after casting it\nis to cast it back to its original type.&sup1;\nIf you try to use it as the cast type, you will crash.\nMaybe not today,\nmaybe not tomorrow, but\n<a HREF=\"http:\/\/blogs.msdn.com\/b\/oldnewthing\/archive\/2004\/01\/19\/60162.aspx\">\nsomeday<\/a>.\n<\/p>\n<p>\nIn this case, the calling convention mismatch resulted in\n<a HREF=\"http:\/\/blogs.msdn.com\/b\/oldnewthing\/archive\/2004\/01\/16\/59415.aspx\">\nthe stack being mismatched when the function returns<\/a>.\nIt looks like earlier versions of Windows managed to hobble\nalong long enough before things got resynchronized\n(by an <code>EBP<\/code> frame restoration, most likely)\nso the damage didn&#8217;t spread very far.\nBut the new version of Windows, possibly one compiled with\nmore aggressive optimizations,\nran into trouble before things resynchronized, and thus occurred\nthe crash.\n<\/p>\n<p>\nThe compiler was yelling at you for a reason.\n<\/p>\n<p>\nIt so happened that the Windows application compatibility team\nhad already encountered this problem in their test labs,\nand a shim had already been developed to auto-correct this\nmistake.\n(Actually, the shim also corrects another mistake they hadn&#8217;t noticed\nyet:\nThey forgot to call <code>Remove&shy;Window&shy;Subclass<\/code>\nwhen they were done.)\n<\/p>\n<p>\n&sup1;I refer here to pointers to static functions.\n<a HREF=\"http:\/\/blogs.msdn.com\/oldnewthing\/archive\/2004\/02\/09\/70002.aspx\">\nPointers to member functions are entirely different animals<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A customer reported an application compatibility bug in Windows. We have some code that manages a Win32 button control. During button creation, we subclass the window by calling Set&shy;Window&shy;Subclass. On the previous version of Windows, the subclass procedure receives the following messages, in order: WM_WINDOWPOSCHANGING WM_NCCALCSIZE WM_WINDOWPOSCHANGED We do not handle any of these messages [&hellip;]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-10723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"<p>A customer reported an application compatibility bug in Windows. We have some code that manages a Win32 button control. During button creation, we subclass the window by calling Set&shy;Window&shy;Subclass. On the previous version of Windows, the subclass procedure receives the following messages, in order: WM_WINDOWPOSCHANGING WM_NCCALCSIZE WM_WINDOWPOSCHANGED We do not handle any of these messages [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/10723","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=10723"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/10723\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=10723"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=10723"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=10723"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}