A customer encountered found that sometimes, their application hung in its clean up code. Here’s a simplified version.<\/p>\n
bool ShuttingDown = false;\r\n\r\nvoid MainThread()\r\n{\r\n DWORD id;\r\n auto hThread = CreateThread(nullptr, 0, WorkerThread,\r\n nullptr, 0, &id); \/\/ succeeds\r\n\r\n BlahBlahBlah(); \/\/ do useful work\r\n\r\n \/\/ Time to clean up. Post an APC to the worker thread\r\n \/\/ to tell it that it's time to go home.\r\n QueueUserAPC(WakeWorker, hThread, 0); \/\/ succeeds\r\n\r\n WaitForSingleObject(hThread, INFINITE); \/\/ hangs\r\n\r\n CloseHandle(hThread);\r\n}\r\n\r\nvoid CALLBACK WakeWorker(ULONG_PTR)\r\n{\r\n ShuttingDown = true;\r\n}\r\n\r\nDWORD CALLBACK WorkerThread(void*)\r\n{\r\n \/\/ Do work until shut down.\r\n do\r\n {\r\n \/\/ All work is posted via APCs.\r\n SleepEx(INFINITE, TRUE);\r\n } while (!ShuttingDown);\r\n\r\n return 0;\r\n}\r\n<\/pre>\nThe idea is that the program has a worker thread to, y’know, do some work. All of the work items are posted via Queue\u00adUser\u00adAPC<\/code>, and the worker thread simply calls Sleep\u00adEx<\/code> alertably, over and over again. Each call to Sleep\u00adEx<\/code> sleeps the thread until an APC is queued, at which point it returns (because the bAlertable<\/code> parameter is TRUE<\/code>).<\/p>\nOne way of looking at this design is that it’s a sneaky way of making the operating system manage your work queue for you. Another way of looking at it is as a single-threaded I\/O completion port.<\/p>\n
Call it what you will.<\/p>\n
Anyway, the problem is that the worker thread is stuck in SleepEx<\/code>, as if it never got the Wake\u00adWorker<\/code> APC that tells it to exit.<\/p>\nBut is that really the problem?<\/p>\n
The customer was able to get some full memory dumps of systems that got into this state, and the telling detail is that the ShuttingDown<\/code> variable was set to true<\/code>. So it wasn’t that the Wake\u00adWorker<\/code> APC never arrived. It totally did arrive and set ShuttingDown<\/code> to true<\/code>. Yet somehow that didn’t wake up the SleepEx<\/code>.<\/p>\nOne of my colleagues offered the possibility that when the code entered the Sleep\u00adEx<\/code> loop, the Wake\u00adWorker<\/code> APC had already run<\/i>. If that’s the case, then the Sleep\u00adEx<\/code> is going to wait for an APC that has already arrived.<\/p>\nI was able to confirm with a test program that this was indeed a possibility.<\/p>\n
DWORD apc = 0;\r\n\r\nvoid CALLBACK WakeWorker(ULONG_PTR)\r\n{\r\n apc = GetCurrentThreadId();\r\n}\r\n\r\nDWORD CALLBACK WorkerThread(void*)\r\n{\r\n if (apc == GetCurrentThreadId()) DebugBreak();\r\n return 0;\r\n}\r\n\r\nint __cdecl main()\r\n{\r\n DWORD id;\r\n while (true)\r\n {\r\n apc = 0;\r\n auto h = CreateThread(nullptr, 0, WorkerThread, nullptr, 0, &id);\r\n QueueUserAPC(WakeWorker, h, 0);\r\n WaitForSingleObject(h, INFINITE);\r\n CloseHandle(h);\r\n Sleep(10);\r\n }\r\n \/\/ notreached\r\n}\r\n<\/pre>\nThis program creates a thread and immediately queues an APC to it. The thread procedure checks if the APC already ran on the same thread, and if so, it breaks into the debugger.<\/p>\n
If you run this program, it breaks immediately. If you set a breakpoint on the Wake\u00adWorker<\/code> function, you’ll see that it does indeed run on the thread before the Worker\u00adThread<\/code> function starts.<\/p>\nFrom the stack trace on that breakpoint, it appears that the kernel processes APCs during the early phases of thread creation, so if you had any queued APCs, they will get processed before the thread procedure even starts.<\/p>\n
So that’s the bug: The ShuttingDown<\/code> flag was already set by the time the thread procedure started, but the code assumes that it can only be set by an APC that is processed by the Sleep\u00adEx<\/code>.<\/p>\nThe fix is simple: Change the loop from a do...while<\/code> to a while<\/code>, so that the test is at the top of the loop instead of at the bottom.<\/p>\nDWORD CALLBACK WorkerThread(void*)\r\n{\r\n \/\/ Do work until shut down.\r\n while (!ShuttingDown)\r\n {\r\n \/\/ All work is posted via APCs.\r\n SleepEx(INFINITE, TRUE);\r\n }\r\n\r\n return 0;\r\n}\r\n<\/pre>\nThe case where this happens is if the Blah\u00adBlah\u00adBlah()<\/code> function returns very quickly, allowing the QueueUserAPC<\/code> to win the race against the Worker\u00adThread<\/code>.<\/p>\n","protected":false},"excerpt":{"rendered":"Or maybe it did?<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-107151","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"
Or maybe it did?<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/107151","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=107151"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/107151\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=107151"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=107151"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=107151"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}