How did we conclude that CcNamespace.dll was the ringleader of a group of DLLs that unloaded prematurely?

When I presented my study of a crash caused by a thread executing from an unloaded third-party DLL, someone asked how I concluded that CcNamespace.dll was the ringleader of the family of related DLLs.

The list of recently-unloaded DLLs is recorded in a circular history.¹ So when you see a bunch of DLLs listed in a row, they were unloaded one after the other.²

00007ff9`6d7c0000 00007ff9`6d80a000   FabrikamContextMenu.dll
00007ff9`115e0000 00007ff9`1172f000   LitWareSync.dll
00007ff9`643d0000 00007ff9`64681000   CcNamespace.dll
00000000`55440000 00000000`5550b000   LibDB_CloudNs_3.dll
00000000`55860000 00000000`55998000   LibNet_CloudNs_3.dll
00000000`557f0000 00000000`5585b000   LibJson_CloudNs_3.dll
00000000`55510000 00000000`557e7000   LibUtils_CloudNs_3.dll
00000000`561a0000 00000000`56238000   MSVCP100.dll
00000000`56240000 00000000`56312000   MSVCR100.dll
00007ff9`85130000 00007ff9`85167000   EhStorShell.dll
00007ff9`3cac0000 00007ff9`3cb61000   wpdshext.dll
00007ff9`78a00000 00007ff9`78a26000   EhStorAPI.dll
00007ff9`686f0000 00007ff9`68754000   PlayToDevice.dll
00007ff9`67110000 00007ff9`6718d000   provsvc.dll

From the similar names, it’s clear that the Lib⟦...⟧.CloudNs.3.dll DLLs are all closely related.

The name CcNamespace.dll sort of lines up, if you imagine that Cc stands for “Contoso Cloud” and the Ns in the other DLL names is short for “Namespace”.

I’m not sure whether the unloaded DLLs list is in forward or reverse chronological order, but it turns out that it’s not important for this analysis: The CcNamespace.dll was either the first or last to unload, and the fact that its name is slightly different from the others adds weight to the theory that it was the linchpin that held together the entire family of DLLs.

Why does having a slightly different name factor into it?

The other DLLs seem to be coming from a library of helper DLLs that this company uses and reuses. The one with a slightly different name is the front man, who is assembling a team of assistants from the other DLLs. It’s like when you attend a wedding reception or other fancy event, you can identify the person in charge of the service staff because they are dressed slightly differently from the others. And you can imagine that if there was some need for the staff to gather together and walk single file, the person in charge would probably be at the front or the end of the line.

Spotting the slightly different name at the start or end of the list is not proof, but it’s very strong evidence. And we were able to verify this theory by installing a copy of Contoso Cloud from the application compatibility library and seeing which DLL was registered as the namespace extension DLL.

¹ The debugger just adds each unloaded DLL to the array, and when it reaches the end of the array, the “where to put the next DLL” wraps around back to the top of the array. Unfortunately, there is no indication in the debugger where the “where to put the next DLL” pointer is.

² Even they are sequential, you don’t know how much time has elapsed between them.