Adventures in application compatibility: The case of the jump into the middle of an instruction from nowhere
A spike of Explorer crashes occurred with the release of a particular Windows Insider build. The crash looked like this:
explorer!SomeRandomInternalFunction+0x7d4: 00007ffe`27b00720 006639 add byte ptr [rsi+39h],ah ds:00000000`0000003a=??
This is most likely a nonsense instruction. There’s no obvious reason to be adding a partial upper register.
It looks like this is either a corrupted instruction pointer or corrupted code, because the first code byte is suspicious zero. And since the second byte is 66h, it looks like an off-by-one, since 66h is not an uncommon initial instruction byte. (It’s the operand size override prefix.) Another clue is that the calling function, not shown here, has no reason to be calling the SomeRandomInternalFunction function, and in fact, checking the alleged caller, it indeed does not call it.
Disassembling around the instruction shows that the instruction pointer is indeed in the middle of an instruction:
00007ffe`27b0071c b820000000 mov eax,20h 00007ffe`27b00721 6639853e020000 cmp word ptr [rbp+23Eh],ax
The instruction pointer is one less than the actual start of the instruction, causing the zero byte at the end of the immediate of the previous instruction to be misinterpreted as the start of the instruction to be executed.
How did we end up in the middle of an instruction?
I did some bulk analysis of all the crash dumps that we received and observed that one third-party DLL was common to all of them. Further investigation shows that this third-party DLL is part of a “shell enhancement” program. This program patches Explorer in order to accomplish its enhancements, and apparently one of its patches went awry.
The interesting thing here was how the program decided where to install its patches, and in particular how it managed to patch an function that was never exported or stored in a vtable: It found the function by contacting the Microsoft symbol server to get the names of all the functions in Explorer and their addresses!
What happened is that we recently made changes to this internal function, and apparently those changes were enough to cause the patcher to go haywire. This is unfortunately a regular occurrence: Whenever a new build goes out, there’s a spike of Explorer crashes because all of these patchers start patching the wrong code, and Explorer starts crashing across all the systems that have these “shell enhancement” programs installed. If you’re really unlucky, their rogue patch crashes something in the Explorer startup path, and users finds themselves stuck with an unusable machine due to Explorer crash loops.
This problem is particularly acute with monthly security patches, because we can’t roll the fix back. That would expose systems to the security issues that the monthly security update was intended to fix. (And now that the fix went out, all the bad guys have reverse-engineered the security issue and are probably hard at work trying to weaponizing it and take advantage of unpatched systems.) We have to hope that enough of the users whose systems are crashing realize that it’s due to the “shell enhancement” program (rather than blaming Windows itself, which is the more likely case), and uninstall or disable the programs in order to get their system working again. Unfortunately, these patchers also cause Windows customer satisfaction numbers to plunge every time an update goes out, particularly among users who don’t realize that the problem was caused by that program their computer-savvy nephew installed for them.
Light
Dark
25 comments
As the OS vendor, MS should have powerful tools available to protect the integrity of its executables. Why doesn’t it do that?
These “Explorer patchers” have no choice, as the normal Explorer in WIndows 11 has lost so much – I’m looking at you, Taskbar!
If MS would only *add* new features and doesn’t remove useful features, all will be well.
Indeed. There wouldn’t be as much of a need if the windows shell team was smarter. As it stands they make the world a worse place each day they exist. If they feel like imposters, it’s because they are.
In fact I actively encourage, support, and celebrate efforts like this. Make their life hell and delay them. If they spend more time on investigating issues like this, they’ll have less time to f**k something else up.
Quite right unfortunately. Since swapping out explorer as the shell doesn’t work anymore we are forced into terrible choices.
I really do want to be able to run cmd.exe or something even more radical as the shell and still open uwp applications; but this just doesn’t work.
Or they could provide appropriate hooking APIs for the need of such extensions.
While I don’t usually like to say this, this time, I have to agree. The shell has lost a lot of features in Windows 11 and this is the only real way to get some useful features back at the moment.
It is MS’s fault these programs exist and are used widely enough to cause big spikes in the statistics. There have been posts made by major news sites about how to use these programs.
This reminds me of MS08-067 where they used strsafe (for NT5) when a simpler fix would have sufficed.
strsafe probably added less than 1KB to code size, but…
Is there an argument for developing something else to bootstrap the basic desktop experience, instead of Explorer — i.e. something that can’t be patched/doesn’t have extension support — and having Explorer be a regular desktop app?
I’m thinking something like the old days of Progman: without it you didn’t have the Windows experience most people were used to, you just had the cyan desktop that didn’t do very much. I guess I’m also thinking of the difference between a Linux distro’s desktop environment vs its window manager.
Would that “solve” the problem of extending Explorer? You could at least sign and/or validate it before any user code ran (maybe via UEFI/SecureBoot?)
(I suppose, given both Server Core and RemoteApps exist — or even the OOBE — at least some of this idea already exists.)
What do you see as the bare-minimum that Windows would need from such a bootstrapper? What additional problems might it introduce?
What would that bootstrapper help with, exactly? What would its responsibiilties be? The actual bootstrapper is userinit.exe, but without explorer.exe, Windows is quite useless (but you can still do things in apps you have open, or launch Task Manager to bring Explorer back). Windows could prevent third-party DLLs from loading, but the very loud community of such patchers would yell “Microsoft is evil, they’re locking down Windows and forcing you to have the taskbar at the botom”. Windows could split explorer.exe into files.exe, taskbar.exe, and desktop.exe, but the patcher community would probably just patch and cause crashes in three processes at once.
“but without explorer.exe, Windows is quite useless (but you can still do things in apps you have open, or launch Task Manager to bring Explorer back)”
Actually, only if Explorer is not there, not if it crashed/hung.
I sometimes have a problem where Edge somehow blocks explorer.exe.
It still runs but clicking on things does nothing.
If you press Ctrl+Alt+Del and select Task Manager then the Ctrl+Alt+Del-screen goes a away but you get no task manager.
But if it was already running and visible, the killing Edge, then Explorer and restarting Explorer gets you a useable PC again.
They could use out of process modern IPC like in Android, macOS/iOS (XPC), Linux (DBUS), instead of in-process COM, with tooling that has hardly changed the last 25 years.
There’s at least one such utility out there that properly enough validates if their patches would work, or blocklist specific Windows builds known to break from it completely. I wish the other very-obvious-from-your-article software would do so too.
Easiest fix for these issues is to dedicate some poor guy at explorer team to be a user of all those shell enchancement (pun intended) softwares.
If this is a serious problem, it could be solved by:
1. having an alternate version of explorer (e.g. called axplorer.exe) which doesn’t load 3rd party shell extensions or any other DLLs that aren’t included with Windows.
2. having userinit (or whatever launches explorer, maybe winlogon?) detect when explorer crashes, and if it crashes more than a particular number of times in a given time frame, load the alternate version and display a dialog to the user explaining why all their shell extensions are not working anymore.
I believe explorer already restarts when it crashes in newer versions of Windows (8 onwards) so part of this solution is already implemented.
If only these programs used normal shell extension mechanisms. But no, they patch the shell by nefarious means.
But how do they get into the explorer.exe address space initially ? Do they load as extension and then break all the rules and start directly patching them ? Maybe you could reveal some tacticss they use and why said tactics are not good, maybe even tell how to do patching correctly ? This could be a new series articles like “the good,bad and ugly of shell patching”
They don’t use the shell extension mechanism to get into the process. They sneak in by nefarious means. Patching is not supported. There is no “correct” way of doing it. Just different levels of bad.
Some would consider having to deal with the primitive COM tooling for shell extensions equally bad.
The taskbar’s also pretty bad. Maybe if it were disconnected from the file browser and from the UWP launcher your patching headaches would be much reduced.
“Just different levels of bad.” I think you said that about me at one point. [grin]
There are no shell extension points for the taskbar/desktop part of Explorer. Deskbars are dead and all other extension points are connected to the file browser and IShellView/IShellFolder.
Did you provide the calls they need to accomplish their mission ?
Not just this article, but far too often when articles such as this are written, about widely used 3rd party tweaks, and the commenting strongly supports the tweak in question, but MS charges forward that “all these enthusiasts are wrong, we will continue our way” is telling of how far out of touch the Win dev team is away from its user base.
It’s not my fault the only way to ungroup icons in the Taskbar is installing sketchy 3rd-party software. I bet someone got a really nice bonus for forcing the taskbar grouping on everyone.