{"id":111079,"date":"2025-04-17T07:00:00","date_gmt":"2025-04-17T14:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/oldnewthing\/?p=111079"},"modified":"2025-04-12T17:26:50","modified_gmt":"2025-04-13T00:26:50","slug":"20250417-00","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20250417-00\/?p=111079\/","title":{"rendered":"The case of the feature flag that didn&#8217;t stay on long enough, part 1"},"content":{"rendered":"<p>There was a crash in a unit test run under the <a title=\"Test Authoring and Execution Framework (TAEF)\" href=\"https:\/\/learn.microsoft.com\/en-us\/windows-hardware\/drivers\/taef\/\"> Test Authoring and Execution Framework<\/a> (TAEF, pronounced like <i>tafe<\/i>). The crashing stack looked like this:<\/p>\n<pre>kernelbase!RaiseFailFastException   \r\nunittests!wil::details::WilDynamicLoadRaiseFailFastException\r\nunittests!wil::details::WilRaiseFailFastException\r\nunittests!wil::details::WilFailFast\r\nunittests!wil::details::ReportFailure_NoReturn&lt;3&gt;\r\nunittests!wil::details::ReportFailure_Base&lt;3,0&gt;\r\nunittests!wil::details::ReportFailure_Hr&lt;3&gt;\r\nunittests!wil::details::in1diag3::_FailFast_Unexpected\r\nunittests!WidgetRouter::GetInstance  \r\nunittests!winrt::Component::implementation::Widget::Close    \r\nunittests!winrt::Component::implementation::Widget::~Widget  \r\nunittests!winrt::impl::heap_implements&lt;\u27e6...\u27e7&gt;::`scalar deleting destructor'   \r\nunittests!winrt::implements&lt;\u27e6...\u27e7&gt;::Release   \r\nunittests!std::vector&lt;winrt::Windows::Foundation::IClosable&gt;::~vector\r\nunittests!std::vector&lt;winrt::Windows::Foundation::IClosable&gt;::clear\r\nunittests!WidgetRouter::~WidgetRouter  \r\nunittests!WidgetRouter::GetInstance'::`2'::`dynamic atexit destructor for 'singleton''\r\nucrtbase!&lt;lambda_\u27e6...\u27e7&gt;::operator    \r\nucrtbase!__crt_seh_guarded_call&lt;int&gt;::operator()&lt;&lt;lambda_\u27e6...\u27e7&gt;&gt;\r\nucrtbase!_execute_onexit_table\r\nucrtbase!__crt_state_management::wrapped_invoke\r\nunittests!dllmain_crt_process_detach  \r\nunittests!dllmain_dispatch    \r\nntdll!LdrpCallInitRoutine   \r\nntdll!LdrpProcessDetachNode \r\nntdll!LdrpUnloadNode    \r\nntdll!LdrpDecrementModuleLoadCountEx    \r\nntdll!LdrUnloadDll  \r\nkernelbase!FreeLibrary  \r\nwex_common!TAEF::Common::Private::ExecutionPeFileData::`scalar deleting destructor' \r\nwex_common!TAEF::Common::PeFile::~PeFile    \r\nte_loaders!WEX::TestExecution::NativeTestFileInstance::`scalar deleting destructor' \r\nte_host!&lt;lambda_\u27e6...\u27e7&gt;::Execute   \r\nte_common!WEX::TestExecution::CommandThread::ExecuteCommandThread   \r\nkernel32!BaseThreadInitThunk\r\nntdll!RtlUserThreadStart\r\n<\/pre>\n<p>The team concluded that the destructor of the <code>Widget<\/code> was running at the conclusion of this function:<\/p>\n<pre>void WidgetTests::BasicTests()\r\n{\r\n  \/\/ Force the feature flag on for the duration of this test\r\n  auto override = std::make_unique&lt;FeatureOverride&gt;(NewFeatureId, true);\r\n\r\n  auto widget = winrt::Component::Widget();\r\n  \u27e6 test the widget in various ways \u27e7\r\n\r\n  \/\/ widget naturally destructs here\r\n}\r\n<\/pre>\n<p>Their conclusion was that the override was being released, and then the widget was destructing, resulting in an assertion failure in <code>WidgetRouter::<wbr \/>GetInstance<\/code>:<\/p>\n<pre>\/* static method *\/\r\nstd::shared_ptr&lt;WidgetRouter&gt;\r\n    WidgetRouter::GetInstance()\r\n{\r\n    assert(FeatureFlags::IsEnabled(NewFeatureId));\r\n\r\n    static std::shared_ptr&lt;WidgetRouter&gt; singleton =\r\n        std::make_shared&lt;WidgetRouter&gt;();\r\n\r\n    return singleton;\r\n}\r\n<\/pre>\n<p>The team theorized that perhaps there was a race condition between the release of the widget and the lifting of the override, and their proposed fix was to release the <code>widget<\/code> explicitly while the override is still in scope.<\/p>\n<pre>void WidgetTests::BasicTests()\r\n{\r\n  \/\/ Force the feature flag on for the duration of this test\r\n  auto override = std::make_unique&lt;FeatureOverride&gt;(NewFeatureId, true);\r\n\r\n  auto widget = winrt::Component::Widget();\r\n  \u27e6 test the widget in various ways \u27e7\r\n\r\n  <span style=\"border: solid 1px currentcolor;\">widget = nullptr;<\/span>\r\n}\r\n<\/pre>\n<p>I commented on the pull request (which had already been completed) that this change has no effect. The rules for C++ say that local variables are destructed in reverse order of construction, so the widget will naturally be released as part of its destruction, and only after that is finished will the override be released when it destructs.<\/p>\n<p>The team replied that they did observe that the problem disappeared after they made their fix, but then it came back.<\/p>\n<p><b>Exercise<\/b>: Explain why the problem went away and then came back.<\/p>\n<p><b>Answer to exercise<\/b>: I suspected that they tested their fix in their team&#8217;s test environment, where the feature is already enabled. The fix works not because it fixed anything but because it was never crashing in their test environment in the first place. The defect tracker, however, doesn&#8217;t know that. It correctly reported that the bug was not being observed in any branches that had received the fix, which was initially only their own test branch. As the fixed merged into other branches, the bug was still not observed, until it finally merged into a branch where their feature was disabled. At that point, the override was actually doing something (changing a feature from disabled to enabled), and that&#8217;s when the crashes started coming in.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Understanding scope and order of destruction.<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-111079","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"<p>Understanding scope and order of destruction.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/111079","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=111079"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/111079\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=111079"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=111079"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=111079"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}