March 24th, 2023

Adventures in application compatibility: The case of the jump into the middle of an instruction from nowhere

A spike of Explorer crashes occurred with the release of a particular Windows Insider build. The crash looked like this:

explorer!SomeRandomInternalFunction+0x7d4:
00007ffe`27b00720 006639          add     byte ptr [rsi+39h],ah ds:00000000`0000003a=??

This is most likely a nonsense instruction. There’s no obvious reason to be adding a partial upper register.

It looks like this is either a corrupted instruction pointer or corrupted code, because the first code byte is suspicious zero. And since the second byte is 66h, it looks like an off-by-one, since 66h is not an uncommon initial instruction byte. (It’s the operand size override prefix.) Another clue is that the calling function, not shown here, has no reason to be calling the Some­Random­Internal­Function function, and in fact, checking the alleged caller, it indeed does not call it.

Disassembling around the instruction shows that the instruction pointer is indeed in the middle of an instruction:

00007ffe`27b0071c b820000000      mov     eax,20h
00007ffe`27b00721 6639853e020000  cmp     word ptr [rbp+23Eh],ax

The instruction pointer is one less than the actual start of the instruction, causing the zero byte at the end of the immediate of the previous instruction to be misinterpreted as the start of the instruction to be executed.

How did we end up in the middle of an instruction?

I did some bulk analysis of all the crash dumps that we received and observed that one third-party DLL was common to all of them. Further investigation shows that this third-party DLL is part of a “shell enhancement” program. This program patches Explorer in order to accomplish its enhancements, and apparently one of its patches went awry.

The interesting thing here was how the program decided where to install its patches, and in particular how it managed to patch an function that was never exported or stored in a vtable: It found the function by contacting the Microsoft symbol server to get the names of all the functions in Explorer and their addresses!

What happened is that we recently made changes to this internal function, and apparently those changes were enough to cause the patcher to go haywire. This is unfortunately a regular occurrence: Whenever a new build goes out, there’s a spike of Explorer crashes because all of these patchers start patching the wrong code, and Explorer starts crashing across all the systems that have these “shell enhancement” programs installed. If you’re really unlucky, their rogue patch crashes something in the Explorer startup path, and users finds themselves stuck with an unusable machine due to Explorer crash loops.

This problem is particularly acute with monthly security patches, because we can’t roll the fix back. That would expose systems to the security issues that the monthly security update was intended to fix. (And now that the fix went out, all the bad guys have reverse-engineered the security issue and are probably hard at work trying to weaponizing it and take advantage of unpatched systems.) We have to hope that enough of the users whose systems are crashing realize that it’s due to the “shell enhancement” program (rather than blaming Windows itself, which is the more likely case), and uninstall or disable the programs in order to get their system working again. Unfortunately, these patchers also cause Windows customer satisfaction numbers to plunge every time an update goes out, particularly among users who don’t realize that the problem was caused by that program their computer-savvy nephew installed for them.

Topics
Other

Author

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

25 comments

Discussion is closed. Login to edit/delete existing comments.

  • Keyboard Bug

    It’s not my fault the only way to ungroup icons in the Taskbar is installing sketchy 3rd-party software. I bet someone got a really nice bonus for forcing the taskbar grouping on everyone.

  • john williamson

    Not just this article, but far too often when articles such as this are written, about widely used 3rd party tweaks, and the commenting strongly supports the tweak in question, but MS charges forward that “all these enthusiasts are wrong, we will continue our way” is telling of how far out of touch the Win dev team is away from its user base.

  • Daniel Roskams

    If this is a serious problem, it could be solved by:

    1. having an alternate version of explorer (e.g. called axplorer.exe) which doesn't load 3rd party shell extensions or any other DLLs that aren't included with Windows.
    2. having userinit (or whatever launches explorer, maybe winlogon?) detect when explorer crashes, and if it crashes more than a particular number of times in a given time frame, load the alternate version and display a dialog to the...

    Read more
    • Raymond ChenMicrosoft employee Author

      If only these programs used normal shell extension mechanisms. But no, they patch the shell by nefarious means.

      • Jules Archinova

        Did you provide the calls they need to accomplish their mission ?

      • skSdnW

        There are no shell extension points for the taskbar/desktop part of Explorer. Deskbars are dead and all other extension points are connected to the file browser and IShellView/IShellFolder.

      • Ismo Salonen

        But how do they get into the explorer.exe address space initially ? Do they load as extension and then break all the rules and start directly patching them ? Maybe you could reveal some tacticss they use and why said tactics are not good, maybe even tell how to do patching correctly ? This could be a new series articles like “the good,bad and ugly of shell patching”

      • Raymond ChenMicrosoft employee Author

        They don’t use the shell extension mechanism to get into the process. They sneak in by nefarious means. Patching is not supported. There is no “correct” way of doing it. Just different levels of bad.

      • Rafael Rivera · Edited

        “Just different levels of bad.” I think you said that about me at one point. [grin]

      • Joshua Hudson

        The taskbar’s also pretty bad. Maybe if it were disconnected from the file browser and from the UWP launcher your patching headaches would be much reduced.

      • Paulo Pinto

        Some would consider having to deal with the primitive COM tooling for shell extensions equally bad.

  • Sigge Mannen

    Easiest fix for these issues is to dedicate some poor guy at explorer team to be a user of all those shell enchancement (pun intended) softwares.

  • Ray Koopa · Edited

    There’s at least one such utility out there that properly enough validates if their patches would work, or blocklist specific Windows builds known to break from it completely. I wish the other very-obvious-from-your-article software would do so too.

  • jimbobmcgee

    Is there an argument for developing something else to bootstrap the basic desktop experience, instead of Explorer -- i.e. something that can't be patched/doesn't have extension support -- and having Explorer be a regular desktop app?

    I'm thinking something like the old days of Progman: without it you didn't have the Windows experience most people were used to, you just had the cyan desktop that didn't do very much. I guess I'm also...

    Read more
    • Paulo Pinto

      They could use out of process modern IPC like in Android, macOS/iOS (XPC), Linux (DBUS), instead of in-process COM, with tooling that has hardly changed the last 25 years.

    • Chris Warrick

      What would that bootstrapper help with, exactly? What would its responsibiilties be? The actual bootstrapper is userinit.exe, but without explorer.exe, Windows is quite useless (but you can still do things in apps you have open, or launch Task Manager to bring Explorer back). Windows could prevent third-party DLLs from loading, but the very loud community of such patchers would yell “Microsoft is evil, they’re locking down Windows and forcing you to have the taskbar at...

      Read more
      • Blubberich · Edited

        "but without explorer.exe, Windows is quite useless (but you can still do things in apps you have open, or launch Task Manager to bring Explorer back)"

        Actually, only if Explorer is not there, not if it crashed/hung.

        I sometimes have a problem where Edge somehow blocks explorer.exe.
        It still runs but clicking on things does nothing.
        If you press Ctrl+Alt+Del and select Task Manager then the Ctrl+Alt+Del-screen goes a away but you get no task manager.
        But...

        Read more
  • Yuhong Bao · Edited

    This reminds me of MS08-067 where they used strsafe (for NT5) when a simpler fix would have sufficed.

    • Yuhong Bao

      strsafe probably added less than 1KB to code size, but…

  • Pavel Yosifovich

    These “Explorer patchers” have no choice, as the normal Explorer in WIndows 11 has lost so much – I’m looking at you, Taskbar!
    If MS would only *add* new features and doesn’t remove useful features, all will be well.

    • Sebastian Kassai

      While I don’t usually like to say this, this time, I have to agree. The shell has lost a lot of features in Windows 11 and this is the only real way to get some useful features back at the moment.

      It is MS’s fault these programs exist and are used widely enough to cause big spikes in the statistics. There have been posts made by major news sites about how to use these programs.

    • Jan Ringoš

      Or they could provide appropriate hooking APIs for the need of such extensions.

    • Joshua Hudson

      Quite right unfortunately. Since swapping out explorer as the shell doesn’t work anymore we are forced into terrible choices.

      I really do want to be able to run cmd.exe or something even more radical as the shell and still open uwp applications; but this just doesn’t work.

    • Roger B

      Indeed. There wouldn't be as much of a need if the windows shell team was smarter. As it stands they make the world a worse place each day they exist. If they feel like imposters, it's because they are.

      In fact I actively encourage, support, and celebrate efforts like this. Make their life hell and delay them. If they spend more time on investigating issues like this, they'll have less time to f**k something else up.

      Read more
  • Brian Dellisanti

    As the OS vendor, MS should have powerful tools available to protect the integrity of its executables. Why doesn’t it do that?