{"id":41043,"date":"2004-01-15T07:00:00","date_gmt":"2004-01-15T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2004\/01\/15\/what-can-go-wrong-when-you-mismatch-the-calling-convention\/"},"modified":"2004-01-15T07:00:00","modified_gmt":"2004-01-15T07:00:00","slug":"what-can-go-wrong-when-you-mismatch-the-calling-convention","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20040115-00\/?p=41043","title":{"rendered":"What can go wrong when you mismatch the calling convention?"},"content":{"rendered":"

Believe it or not, calling conventions is one of the things that\nprograms frequently get wrong. The compiler yells at you when\nyou mismatch a calling convention, but lazy programmers will just\nstick a cast in there to get the compiler to “shut up already”.<\/p>\n

\nAnd then Windows is stuck having to support your buggy code forever.\n<\/p>\n

\n
The window procedure<\/dt>\n
\n

\nSo many people misdeclare their window procedures (usually by\ndeclaring them as __cdecl instead of __stdcall),\nthat the function that dispatches messages to window procedures\ncontains extra protection\nto detect incorrectly-declared window procedures and perform the\nappropriate fixup. This is the source of the mysterious 0xdcbaabcd\non the stack. The function that dispatches messages to window\nprocedures checks whether this value is on the stack in the correct\nplace. If not, then it checks whether the window procedure popped\none dword too much off the stack (if so, it fixes up the stack;\nI have no idea how this messed up a window procedure could have existed),\nor whether the window procedure was mistakenly declared as __cdecl\ninstead of __stdcall (if so, it pops the parameters off the stack that\nthe window procedure was supposed to do).\n<\/p>\n<\/dd>\n

DirectX callbacks<\/dt>\n
\n

\nMany DirectX functions use callbacks, and people once again misdeclared\ntheir callbacks as __cdecl instead of __stdcall, so the DirectX\nenumerators have to do special stack cleanup for those bad functions.\n<\/p>\n<\/dd>\n

IShellFolder::CreateViewObject<\/dt>\n
\n

\nI remember there was one program that decided to declare their\nCreateViewWindow function incorrectly, and somehow they managed\nto trick the compiler into accepting it!\n<\/p>\n

\nclass BuggyFolder : public IShellFolder ... {\n ...\n \/\/ wrong function signature!\n HRESULT CreateViewObject(HWND hwnd) { return S_OK; }\n}\n<\/pre>\n
\nNot only did they get the function signature wrong, they\nreturned S_OK even though they failed to do anything!\nI had to add extra code to clean up the stack after calling this\nfunction, as well as verify that the return value wasn’t a lie.\n<\/p>\n
Rundll32.exe entry points<\/dt>\n
\n
\n\nThe function signature required for functions called by rundll32.exe\nis documented in this Knowledge Base article<\/a>.\nThat hasn’t stopped people from using rundll32 to call random functions\nthat weren’t designed to be called by rundll32,\n\nlike user32 LockWorkStation<\/a> or\n\nuser32 ExitWindowsEx<\/a>.\n<\/p>\n
\nLet’s walk through what happens when you try to use rundll32.exe\nto call a function like ExitWindowsEx:\n<\/p>\n
\nThe rundll32.exe program parses its command line and calls the\nExitWindowsEx function on the assumption that the function is written\nlike this:<\/p>\n
\nvoid CALLBACK ExitWindowsEx(HWND hwnd, HINSTANCE hinst,\n       LPSTR pszCmdLine, int nCmdShow);\n<\/pre>\n
But it isn’t. The actual function signature for ExitWindowsEx is<\/p>\n
\nBOOL WINAPI ExitWindowsEx(UINT uFlags, DWORD dwReserved);\n<\/pre>\n
What happens? Well, on entry to ExitWindowsEx, the stack\nlooks like this:\n<\/p>\n\n\n\n\n\n\n\n\n
.. rest of stack ..<\/td>\n<\/td>\n<\/tr>\n
nCmdShow<\/td>\n<\/td>\n<\/tr>\n
pszCmdLine<\/td>\n<\/td>\n<\/tr>\n
hinst<\/td>\n<\/td>\n<\/tr>\n
hwnd<\/td>\n<\/td>\n<\/tr>\n
return address<\/td>\n<- ESP<\/td>\n<\/tr>\n<\/table>\n
\nHowever, the function is expecting to see<\/p>\n\n\n\n\n\n\n
.. rest of stack ..<\/td>\n<\/td>\n<\/tr>\n
dwReserved<\/td>\n<\/td>\n<\/tr>\n
uFlags<\/td>\n<\/td>\n<\/tr>\n
return address<\/td>\n<- ESP<\/td>\n<\/tr>\n<\/table>\n
\nWhat happens? The hwnd passed by rundll32.exe gets misinterpreted\nas uFlags and the hinst gets misinterpreted as dwReserved.\nSince window handles are pseudorandom, you end up passing random\nflags to ExitWindowsEx.  Maybe today it’s EWX_LOGOFF, tomorrow it’s\nEWX_FORCE, the next time it might be EWX_POWEROFF.\n<\/p>\n
\nNow suppose that the function manages to return. (For example,\nthe exit fails.)  The ExitWindowsEx function cleans two parameters\noff the stack, unaware that it was passed four. The resulting stack\nis<\/p>\n\n\n\n\n\n
.. rest of stack ..<\/td>\n<\/td>\n<\/tr>\n
nCmdShow<\/td>\n(garbage not cleaned up)<\/td>\n<\/tr>\n
pszCmdLine<\/td>\n<- ESP (garbage not cleaned up)<\/td>\n<\/tr>\n<\/table>\n
Now the stack is corrupted and really fun things happen.\nFor example, suppose the thing at “.. rest of the stack ..” is\na return address.  Well, the original code is going to execute\na “return” instruction to return through that return address,\nbut with this corrupted stack, the “return” instruction will\ninstead return to a command line and attempt to execute it as if\nit were code.\n<\/p>\n<\/dd>\n
Random custom functions<\/dt>\n
\n\nAn anonymous commenter exported a function as __cdecl but treated it as if\nit were __stdcall<\/a>.  This will seem to work, but on return, the stack\nwill be corrupted (because the caller is expecting a __stdcall function\nthat cleans the stack, but what it gets is a __cdecl funcion that doesn’t),\nand bad things will happen as a result.\n<\/dd>\n<\/dl>\n
\nOkay, enough with the examples; I think you get the point.\nHere are some questions I’m sure you’re asking:\n<\/p>\n
\n
Why doesn’t the compiler catch all these errors?<\/dt>\n
\n
\nIt does. (Well, not the rundll32 one.)\nBut people have gotten into the habit of just inserting the\nfunction cast to get the compiler to shut up.\n<\/p>\n
\nHere’s\na random example I found<\/a>:<\/p>\n
\nLRESULT CALLBACK DlgProc(HWND hWnd, UINT Msg,\n   WPARAM wParam, LPARAM lParam);\n<\/pre>\n
\nThis is the incorrect function signature for a dialog procedure.\nThe\ncorrect signature<\/a> is<\/p>\n
\nINT_PTR CALLBACK DialogProc(HWND hwndDlg, UINT uMsg,\n    WPARAM wParam, LPARAM lParam);\n<\/pre>\n<\/p>\n
\nYou start with<\/p>\n
\nDialogBox(hInst, MAKEINTRESOURCE(IDD_CONTROLS_DLG),\n          hWnd, DlgProc);\n<\/pre>\n
but the compiler rightly spits out the error message<\/p>\n
\nerror C2664: 'DialogBoxParamA' : cannot convert parameter 4\nfrom 'LRESULT (HWND,UINT,WPARAM,LPARAM)' to 'DLGPROC'\n<\/pre>\n
so you fix it by slapping a cast in to make the compiler\nshut up:<\/p>\n
\nDialogBox(hInst, MAKEINTRESOURCE(IDD_CONTROLS_DLG),\n          hWnd, reinterpret_cast<DLGPROC><\/font>(DlgProc));\n<\/pre>\n<\/p>\n
\n“Aw, come on, who would be so stupid as to insert a cast to\nmake an error go away without actually fixing the error?”\n<\/p>\n
\nApparently everyone.\n<\/p>\n
\nI stumbled across\n\nthis page that does exactly the same\nthing<\/a>,\n\nand this one in German which gets not only the return value wrong,\nbut also misdeclares the third and fourth parameters<\/a>,\n\nand this one in Japanese<\/a>.\n\nIt’s as easy to fix (incorrectly) as 1-2-3<\/a>.\n<\/p>\n<\/dd>\n
How did programs with these bugs ever work at all?\nCertainly these programs worked to some degree or people would have\nnoticed and fixed the bug.\nHow can the program survive a corrupted stack?<\/dt>\n
\n
\nI’ll answer this question tomorrow.\n<\/dd>\n<\/dl>\n","protected":false},"excerpt":{"rendered":"
Believe it or not, calling conventions is one of the things that programs frequently get wrong. The compiler yells at you when you mismatch a calling convention, but lazy programmers will just stick a cast in there to get the compiler to “shut up already”. And then Windows is stuck having to support your buggy […]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-41043","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Believe it or not, calling conventions is one of the things that programs frequently get wrong. The compiler yells at you when you mismatch a calling convention, but lazy programmers will just stick a cast in there to get the compiler to “shut up already”. And then Windows is stuck having to support your buggy […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/41043","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=41043"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/41043\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=41043"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=41043"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=41043"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}