{"id":95345,"date":"2017-02-03T07:00:00","date_gmt":"2017-02-03T22:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/?p=95345"},"modified":"2019-03-13T01:05:36","modified_gmt":"2019-03-13T08:05:36","slug":"20170203-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20170203-00\/?p=95345","title":{"rendered":"Why am I getting a crash at shutdown inside the thread pool?"},"content":{"rendered":"<p>A customer reported a crash in WinHTTP when their application shuts down a WebSocket. Specifically, it occurs when one of their DLL&#8217;s global objects is being destructed. <\/p>\n<p>The customer sent us a redacted call stack: <\/p>\n<pre>\n00a5e11c 7753ebbe ntdll!KiFastSystemCallRet\n00a5e120 77581174 ntdll!NtAlpcSendWaitReceivePort+0xa\n00a5e1d0 7758078a ntdll!SendMessageToWERService+0x14d\n00a5ecc0 77580c10 ntdll!ReportExceptionInternal+0xde\n00a5f118 7758085b ntdll!RtlReportExceptionEx+0x379\n00a5f170 775a74dc ntdll!RtlReportException+0x9b\n00a5f180 77541454 ntdll!TppRaiseInvalidParameter+0x51\n00a5f194 77540ddd ntdll!_EH4_CallFilterFunc+0x12\n00a5f1bc 77544d33 ntdll!_except_handler4_common+0x8d\n00a5f1dc 775508d2 ntdll!_except_handler4+0x20\n00a5f200 775508a4 ntdll!ExecuteHandler2+0x26\n00a5f2c8 7753f477 ntdll!ExecuteHandler+0x24\n00a5f2c8 775a74c2 ntdll!KiUserExceptionDispatcher+0xf\n00a5f660 7755ddb0 ntdll!TppRaiseInvalidParameter+0x37\n00a5f66c 774ecdd2 ntdll!TppTimerpValidateTimer+0x6e1a2\n00a5f690 757ddadb ntdll!TpSetTimerEx+0x1b\n00a5f6b8 757c646d WINHTTP!HTTP_THREAD_POOL::SetTimer+0x42\n00a5f6f0 757c6070 WINHTTP!WEB_SOCKET_HANDLE_OBJECT::Close+0x1bb\n00a5f754 69699832 WINHTTP!WinHttpWebSocketClose+0x9c\n...\n global atexit call being made here\n...\n00a5f814 696d1f7d XXXXXX!_CRT_INIT+0xaa\n00a5f874 7753cd4e XXXXXX!__DllMainCRTStartup+0x1ee\n00a5f894 77505525 ntdll!LdrxCallInitRoutine+0x16\n00a5f8e4 775057cb ntdll!LdrpCallInitRoutine+0x43\n00a5f97c 77518e3f ntdll!LdrShutdownProcess+0x101\n00a5f990 77065736 ntdll!RtlExitUserProcess+0x63\n00a5f99c 77065471 msvcrt!__crtExitProcess+0x17\n00a5f9e0 77065715 msvcrt!doexit+0x10a\n00a5f9f4 00be2369 msvcrt!exit+0x11\n00a5fa2c 7752b2dd contoso!__wmainCRTStartup+0x114\n00a5fa70 7752b2a7 ntdll!__RtlUserThreadStart+0x2f\n00a5fa80 00000000 ntdll!_RtlUserThreadStart+0x1b\n<\/pre>\n<p>The customer concluded, &#8220;We have some ideas that may work around the issue by using <code>WINHTTP_OPTION_WEB_SOCKET_CLOSE_TIMEOUT<\/code> to avoid the close timeout, but we&#8217;d like confirmation as to whether this will actually solve the problem.&#8221; <\/p>\n<p>Okay, first let&#8217;s understand the problem, then we can look at possible solutions. <\/p>\n<p>The customer has a DLL with a global object, and <a HREF=\"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/20141017-00\/?p=43823\">as we learned some time ago<\/a>, global objects in DLLs are destructed as part of <code>DLL_PROCESS_DETACH<\/code>. The problem is that the thread pool has already shut down by the time this DLL gets around to destroying global objects. We know this because <a HREF=\"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/20070503-00\/?p=27003\">one of the first steps in process termination is terminating all but one of the threads<\/a>. A thread pool without any threads is not really a thread pool any more. <\/p>\n<p>At process termination, the thread pool is electrified. Any attempt to schedule new work on the thread pool will result in an immediate crash. In this case, the problem is that the customer&#8217;s DLL is closing a WinHTTP WebSocket, and one of the things that WinHTTP does when it closes a WebSocket is to schedule a thread pool timer so it can abort the close handshake if it takes too long. <\/p>\n<p>Okay, so the chain of events goes like this: Thread pool gets electrified, then the DLL starts destructing its objects, and one of the objects tries to close a WebSocket, and closing the WebSocket creates a thread pool timer, but the thread pool is electrified, so the process crashes. <\/p>\n<p>Okay, now that we understand the problem, let&#8217;s look for solutions. <\/p>\n<p>The customer&#8217;s proposed workaround is to use <code>WINHTTP_OPTION_WEB_SOCKET_CLOSE_TIMEOUT<\/code> to set the timeout to <code>INFINITE<\/code>. This tells WinHTTP to let the close operation take as long as it wants, which means that it doesn&#8217;t bother creating a thread pool timer to abort a close operation that is taking too long (because you said that there&#8217;s no such thing as &#8220;too long&#8221;). <\/p>\n<p>That solves the proximate problem, but really this is just playing whack-a-mole. You may be able to get rid of this crash caused by closing a WinHTTP WebSocket, but this may merely expose some other object that is also using the thread pool at destruction, and you&#8217;re going to have to go through all this analysis again and look for a way to get that other object to avoid the thread pool at process termination. <\/p>\n<p>The best solution is to try to <a HREF=\"https:\/\/google.github.io\/styleguide\/cppguide.html#Static_and_Global_Variables\">get rid of the global variables in the first place<\/a>. If you can&#8217;t do that, then you at least want to avoid running the destructors at process termination. There are a few ways of accomplishing this: <\/p>\n<ul>\n<li>Clean up the global variables explicitly     prior to process termination.     The destructors will run at     <code>DLL_PROCESS_DETACH<\/code>,     but since you already released the resources,     the destructors won&#8217;t do anything.<\/li>\n<li>Neuter the global variables in     <code>DLL_PROCESS_DETACH<\/code>     if the reason for the notification is that the     process is terminating.     That way, when their destructors run,     they won&#8217;t do anything.<\/li>\n<li>A special case of the previous item is to set     a flag in     <code>DLL_PROCESS_DETACH<\/code>     if the reason for the notification is that the     process is terminating.     Have the destructors check the flag and do nothing     if the flag is set.<\/li>\n<\/ul>\n<p>The point is that you don&#8217;t want to do any cleanup at process termination, because the process has already stopped providing services, and lots of things may be electrified. You just want to let the process terminate and stay out of its way. <\/p>\n<p><b>Exercise<\/b>: By a startling coincidence, the day I wrote this blog entry, this question arrived from another customer. Use what you know to diagnose the customer&#8217;s problem. (In particular, why is the problem sporadic?) <\/p>\n<blockquote CLASS=\"q\">\n<p>We are using a C++ wrapper around Win32 timers. During object destruction, we deactivate the timer by following <a HREF=\"https:\/\/msdn.microsoft.com\/library\/windows\/desktop\/ms686271(v=vs.85).aspx\">the recommended pattern<\/a>: <code>::Set&shy;Threadpool&shy;Timer(this-&gt;GetHandle(), nullptr, 0, 0);<\/code> This works fine, but in some rare scenarios, we encounter this crash. <\/p>\n<pre>\nntdll!ZwWaitForMultipleObjects+0xa\nntdll!RtlReportExceptionEx+0x452\nntdll!RtlReportException+0xbc\nntdll!TppReportExceptionFilter+0x16\nntdll!TppRaiseInvalidParameter$filt$0+0xe\nntdll!__C_specific_handler+0x96\nntdll!__GSHandlerCheck_SEH+0x76\nntdll!RtlpExecuteHandlerForException+0xd\nntdll!RtlDispatchException+0x197\nntdll!RtlRaiseException+0x18d\nntdll!TppRaiseInvalidParameter+0x48\nntdll!TppTimerpValidateTimer+0x6eb93\nntdll!TpSetTimerEx+0x33\ncontoso!WinAPI::ThreadPool::Timer&lt;...&gt;::Reset+0x12\ncontoso!WinAPI::ThreadPool::Timer&lt;...&gt;::{dtor}+0x12\ncontoso!std::default_delete&lt;WinAPI::ThreadPool::Timer&lt;...&gt;&gt;::operator()+0x12\ncontoso!std::unique_ptr&lt;WinAPI::ThreadPool::Timer&lt;...&gt;, ...&gt;::reset+0x23\ncontoso!Contoso::SharedMemoryCache::~SharedMemoryCache+0x57\ncontoso!Contoso::SharedMemoryCache::`scalar deleting destructor'+0x14\ncontoso!std::_Ref_count_base::_Decref+0x17\ncontoso!std::_Ptr_base&lt;...&gt;::_Decref+0x20\ncontoso!std::shared_ptr&lt;...&gt;::{dtor}+0x20\ncontoso!std::tuple&lt;...&gt;::~tuple&lt;...&gt;+0x49\ncontoso!`dynamic atexit destructor for 'Extension::s_extension''+0x23\nucrtbase!&lt;lambda_275893d493268fdec8709772e3fcec0e&gt;::operator()+0xb7\nucrtbase!__crt_seh_guarded_call&lt;int&gt;::operator()&lt;...&gt;+0x3b\nucrtbase!__acrt_lock_and_call+0x1e\nucrtbase!_execute_onexit_table+0x31\ncontoso!dllmain_crt_process_detach+0x4e\ncontoso!dllmain_dispatch+0xd3\nntdll!LdrpCallInitRoutine+0x4c\nntdll!LdrShutdownProcess+0x142\nntdll!RtlExitUserProcess+0x98\nkernel32!ExitProcessImplementation+0xa\ncontososerver!ControlSignalHandler::HandleControlSignal+0x68\nKERNELBASE!CtrlRoutine+0xb3\nkernel32!BaseThreadInitThunk+0x22\nntdll!RtlUserThreadStart+0x34\n<\/pre>\n<p>Any pointers would be appreciated. <\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Stop talking to a dead thread pool.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-95345","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"<p>Stop talking to a dead thread pool.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/95345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=95345"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/95345\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=95345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=95345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=95345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}