Some time ago, I was enlisted to help debug an elusive deadlock. Studying a sampling of process memory dumps led to the conclusion that a critical section had been orphaned. Sometimes, the thread that owned the critical section had already exited, but sometimes the thread was still running, but it was running code completely unrelated to the critical section. It was as if the code that acquired the critical section had simply forgotten to release it before returning. <\/p>\n
The thing is, all attempts to acquire the critical section were managed by an RAII type, so there should be no way that the critical section could have been forgotten. And yet it was. <\/p>\n
When would the destructor for an RAII object by bypassed? One possibility is that somebody did an On all platforms other than x86, exception unwind information is kept in tables in a rarely-used portion of the image, so that we don’t waste memory on exception unwind information until an exception actually occurs: When an exception occurs, the system pages in the unwind tables and does a table lookup to see which unwind handler should run. But on x86, the exception unwind state is maintained manually in the code. This is a bad thing for x86 performance, but a good thing for getting inside the head of the compiler. <\/p>\n Bonus reading: Unwinding the Stack: Exploring How C++ Exceptions Work on Windows<\/a>. The unwind checkpoint is a 32-bit value, usually stored at There are four functions that enter the critical section in question. The code that does so looks like this:<\/p>\n Finding the exact point where the guards are created is made easier with the assistance of the Okay, so the debugger found a line of assembly that mentions We see that immediately after acquiring the lock, the code updates Exercise<\/b>: I said that the unwind state is recorded in a 32-bit value stored at The lock is acquired again later in that same function, so we’ll search some more. If you leave off the second parameter to the Okay, so this lock guard is also marked for unwinding. <\/p>\n The next function that uses the critical section is Hm, this function doesn’t create an unwind checkpoint after taking the lock. This means that the compiler believes that no exception can occur between the point the guard is created and the next thing that would require updating the unwind checkpoint (in our case, that would be the point the lock is destructed). <\/p>\n We repeat this analysis with the other two functions. One of them creates an unwind checkpoint; the other doesn’t. <\/p>\n Why does the compiler believe that no exceptions can occur in the guarded block? Well, inside the block it calls Observe that the This code was compiled before the Microsoft C++ compiler added the My theory is that the Digging into the two objects wrapped inside the Exercise<\/b>: The first was really an The Now you see the problem. If the As is often the case, it’s usually a lot easier to find something once you know what you’re looking for. The team dug into its telemetry to see that, yes indeed, the systems that encountered the problem had also thrown an exception from Now they could work on fixing the problem: Fix the destructor so it doesn’t throw any exceptions. In this specific case, the exception was thrown because they were deactivating an object that hadn’t been fully activated. Since cleanup functions cannot fail<\/a>, the best you can do is to just soldier on and clean up as much as you can<\/a>. <\/p>\n","protected":false},"excerpt":{"rendered":" Rolling up the debugging sleeves.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-100585","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":" Rolling up the debugging sleeves.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/100585","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=100585"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/100585\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=100585"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=100585"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=100585"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}ExitThread<\/code> or (horrors) TerminateThread<\/code>. But this doesn’t match the evidence, because as noted above, in some of the crash dumps, the critical section owner is still alive and running, but unaware that it owns the critical section. <\/p>\n
— James McNellis<\/a>, CppCon 2018 <\/p><\/blockquote>\n[ebp-4]<\/code>. The compiler uses it to keep track of what needs to get unwound if an exception occurs. If the compiler can deduce that no exception can occur between two checkpoints, then it can optimize out the first checkpoint. <\/p>\n\n{\n auto guard = SystemChangeListenerCS.Lock();\n ... some code ...\n} \/\/ guard destructor releases the lock\n<\/pre>\n#<\/code> debugger command, which means “Disassemble until you see this string in the disassembly.” <\/p>\n\n0:000> #SystemChangeListenerCS SystemChangeListenerThreadProc\nSystemChangeListenerThreadProc+0x7c:\n1003319c mov ecx,offset SystemChangeListenerCS (100b861c)\n0:000>\n<\/pre>\n
SystemChangeListenerCS<\/code>. Let’s look to see whether there is an unwind checkpoint after the lock is taken. <\/p>\n\n0:000> u 1003319c\nChangeMonitorThreadProc+0x7c:\n1003319c mov ecx,offset contoso!SystemChangeListenerCS (100b861c)\n100331a1 push eax\n100331a2 call Microsoft::WRL::Wrappers::CriticalSection::Lock (1002a863)\n100331a7 mov byte ptr [ebp-4],5\n<\/pre>\n
[ebp-4]<\/code> to remember that it needs to destruct the lock guard in case an exception occurs. <\/p>\n[ebp-4]<\/code>, but the code here updates only a byte. Why only a byte? <\/p>\n#<\/code> command, it continues searching where the previous search left off. <\/p>\n\n0:000> #SystemChangeListenerCS\nSystemChangeListenerThreadProc+0x487:\n100335a7 mov ecx,offset contoso!SystemChangeListenerCS (100b861c)\n0:000> u 100335a7\ncontoso!SystemChangeListenerThreadProc+0x487:\n100335a7 mov ecx,offset contoso!SystemChangeListenerCS (100b861c)\n100335ac push eax\n100335ad call Microsoft::WRL::Wrappers::CriticalSection::Lock (1002a863)\n100335b2 mov byte ptr [ebp-4],0Dh\n<\/pre>\n
ResetWidgets<\/code>. <\/p>\n\n0:000> #SystemChangeListenerCS ResetWidgets\nResetWidgets+0x133:\n10033fcc mov ecx,offset SystemChangeListenerCS (100b861c)\n0:000> u 10033fcc l4\nResetWidgets+0x133:\n10033fcc mov ecx,offset SystemChangeListenerCS (100b861c)\n10033fd1 push eax\n10033fd2 call Microsoft::WRL::Wrappers::CriticalSection::Lock (1002a863)\n10033fd7 call Microsoft::WRL::ComPtr<IStream>::Reset (10039932)\n10033fdc call Microsoft::WRL::ComPtr<Widget>::Reset (10039142)\n10033fe1 cmp dword ptr [ebp-4Ch],0\n10033fe5 je ResetWidgets+0x157 (10033ff0)\n10033fe7 push dword ptr [ebp-4Ch]\n<\/pre>\n
ComPtr::<\/code>Reset<\/code><\/a> twice, and it does some other stuff. The Reset<\/code> method is declared like this: <\/p>\n\ntemplate<typename T>\nclass ComPtr {\nunsigned long Reset() { return InternalRelease(); }\nunsigned long InternalRelease() throw() { ... }\n...\n};\n<\/pre>\nInternalRelease<\/code> method<\/a> uses the deprecated throw()<\/code> specifier, which says that the method never throws an exception. The compiler then inferred that the Reset<\/code> method also never throws an exception, since it does nothing that could result in an exception. <\/p>\n\/std:C++17<\/code> switch, so it uses the old rules for the throw()<\/code> specifier<\/a>, which for the Microsoft C++ compiler boils down to “I’m trusting you never to throw an exception.” <\/p>\nReset<\/code> actually did throw an exception. Since the compiler didn’t create an unwind checkpoint, the lock guard did not get unwound. The exception was caught higher up the stack, so the process didn’t crash. <\/p>\nComPtr<\/code> revealed that the first one was a WidgetMonitor<\/code> object. <\/p>\nIWidgetMonitor<\/code> interface, so why did it disassemble as ComPtr<IStream><\/code>? <\/p>\nWidgetMonitor<\/code>‘s destructor went like this: <\/p>\n\nWidgetMonitor::~WidgetMonitor()\n{\n Uninitialize();\n}\n\nvoid WidgetMonitor::Uninitialize()\n{\n blah blah;\n ThrowIfFailed(m_monitor.Deactivate());\n blah blah;\n ThrowIfFailed(m_monitor.Disconnect());\n blah blah;\n}\n<\/pre>\nUninitialize<\/code> method throws an exception, the exception will propagate out of the destructor. (This code is so old that it predates C++11’s rule that destructors are noexcept<\/code> by default where possible.) And then it will propagate out of ComPtr::<\/code>InternalRelease<\/code>, and then out of ComPtr::<\/code>Reset<\/code>, and then out of ResetWidgets<\/code>. And unwinding out of ResetWidgets<\/code> will not run the lock guard’s destructor because the compiler assumed that no exception could be thrown, thanks to the throw()<\/code> specifier on the ComPtr::<\/code>InternalRelease<\/code> method. <\/p>\nWidgetMonitor::<\/code>Uninitialize<\/code>, thus confirming the theory. <\/p>\n