{"id":105639,"date":"2021-09-02T07:00:00","date_gmt":"2021-09-02T14:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=105639"},"modified":"2021-09-02T06:32:13","modified_gmt":"2021-09-02T13:32:13","slug":"20210902-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20210902-00\/?p=105639","title":{"rendered":"Adventures in application compatibility: The cost of forgetting to specify a calling convention"},"content":{"rendered":"

We saw last time that the Windows header files sometimes look at world through __stdcall<\/code>-colored glasses<\/a>, and that causes problems when the header file fails to specify an explicit calling convention.<\/p>\n

The developers of one particular component made the mistake of omitting an explicit calling convention for one of their callback function pointer types, but it didn’t cause any immediate problems. Consumers who compiled with __cdecl<\/code> as the default calling convention passed a __cdecl<\/code> function pointer, but things happened to work out okay<\/a>.<\/p>\n

However, people reported that after installing a sevicing update, some programs that used that component started crashing. The reason is that the servicing update altered the code generation, and now the misplaced stack pointer started causing problems.<\/p>\n

What we have here is a confluence of multiple mistakes. The feature team authored their header file incorrectly, failing to specify an explicit calling convention. This led to customers consuming the header file incorrectly, and passing callback function pointers that used the __cdecl<\/code> calling convention instead of __stdcall<\/code>.<\/p>\n

Now the application compatibility adventure begins.<\/p>\n

In addition to fixing the header files to be explicit about the calling convention (to prevent the problem from spreading), the component has to be modified so that it can be used with either<\/i> calling convention.<\/p>\n

declspec(naked) declspec(noinline)\r\nvoid CALLBACK\r\nWrapCallbackWithESPFix(WIDGETFILTERPROC filter, int a, int b)\r\n{\r\n    __asm\r\n    {\r\n        mov     edi, edi                ; hotpatch stub\r\n        push    ebp                     ; establish stack frame\r\n        mov     ebp, esp\r\n        \r\n        push    b\r\n        push    a\r\n#if _CONTROL_FLOW_GUARD\r\n        mov     ecx, filter             ; call target\r\n        call    [__guard_check_icall_fptr]\r\n        call    ecx\r\n#else\r\n        call    filter                  ; make the call\r\n#endif\r\n\r\n        ; restore esp if the callee mismanaged it due to wrong calling convention\r\n        mov     esp, ebp\r\n        pop     ebp\r\n        ret     12\r\n    }\r\n}\r\n<\/pre>\n
It so happens that this workaround didn’t hang around indefinitely. The component in question has a very small audience, and in particular, only one of the clients was encountering this problem. That customer made a fix for their program and deployed it via their update channel. The workaround was removed a little less than a year later.<\/p>\n
Bonus reading<\/b>:  Throwing garbage on the sidewalk: The sad history of the rundll32 program<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Cleaning up behind mistakes of the past.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-105639","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Cleaning up behind mistakes of the past.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/105639","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=105639"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/105639\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=105639"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=105639"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=105639"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}