January 13th, 2023

Adventures in application compatibility: Querying for an internal interface

An application compatibility report came in for a program that crashed during installation. The primary installer created a temporary directory and then ran a secondary installer from that temporary directory. So far, that’s standard behavior to avoid operating from a disease-ridden hot tub. What made it more frustrating to debug is that the secondary installer in turn runs a tertiary installer, and it was the tertiary installer that was crashing. (At least it stopped at the tertiary installer. Otherwise it’s installers all the way down.)

Some debugging of the tertiary installer revealed that it was trying to pin itself to the taskbar automatically on install. This is one of those things that installers love to do, probably second only to silently taking over popular file extensions. This particularly program’s approach was to find the vtable for the internal interface that Explorer uses to pin items to the taskbar, and then call the fifth function in the vtable.

The application compatibility issue was that we changed the signature of the fifth function in the vtable, so they were calling the function with the old signature, and the implementation was therefore misinterpreting the parameters, and as a bonus insult, it also corrupted the stack because the number of parameters also changed.

The solution was to restore the function to its original signature and have it just return without doing anything, decorated with comments to ensure that the method stays at the same slot in the vtable for eternity. Meanwhile, the new functionality was moved to a different location. (I’m not telling where.)

Topics
Other

Author

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

18 comments

Discussion is closed. Login to edit/delete existing comments.

  • Frédéric B.

    I bet someone got a really nice bonus for that “feature”.

  • Sigge Mannen

    This is my favourite kind of oldnewthing story. What i always wonder though is why Microsoft don’t open up these “forbidden” APIs, but embeds user confirmation dialogs in them? Sure, it’s bad sport to mix UI and functions but still.

    Meanwhile, the new functionality was moved to a different location. (I’m not telling where.)

    It was probably moved to 6th vtable entry.

    • Andy Cadley

      Because developers abuse them, no matter how user hostile that is. Want the user to say Yes to pinning your app?

      While (app_is_not_pinned)
      {
         call request_app_to_be_pinned();
      }

      And then just spam the user with acceptance dialogue until they just click Yes. There are already enough out there that guide users through a process to change global settings and keep pestering if you don’t.

      • George Tokmaji · Edited

        You wouldn’t even need to pester the user like that – unless the dialog is isolated like the UAC dialog, the application can just click the Yes button itself via a window message or UI Automation. Or the application could just patch the internal function so that it doesn’t open a dialog in the first place, or hook CreateDialog or whatever API call gets used to create the dialog.

      • Sigge Mannen

        Yeah, but for such hostile apps it’s better if they patch out known APIs than to patch some random vtables by scanning memory

  • Ray Koopa

    I thought pinning stuff to the Win7+ taskbar (“Superbar”) was never meant to be controlled programmatically? Like that’s up to the user to decide what is pinned, as with pinned apps in the old Win7+ start menu? Are some applications so important that Microsoft ignores their purposefully set limitations and fixes compatibility if these apps work around them – or do I just badly remember the set limitations…?

    • Neil Rashbrook

      I know of at least one program that at least only offers to programmatically pin itself to the taskbar for you, so still leaving it up to the user (I declined the pin). Its approach to pinning itself is at least somewhat less hacky than the installer in the article.

    • Raymond ChenMicrosoft employee Author

      Notice that the “fix” changes the function to a nop. The limitation is still in force.

      • Ray Koopa

        Oh, I missed that detail in my rage over that app. That kinda makes more sense. Yet I’d personally be unsatisfied without such programs crashing from doing this bogus and code getting yucky because of them. But yeah, we know who’d get the blame for the crash eventually…

  • Gunnar Dalsnes

    Was it not possible to shim/patch the installer? Or is that much harder than changing explorer?

    • Henke37

      Rule of multiple offenders. Especially with hacky code like this that just begs to be shared around.

  • skSdnW

    The shell team have also decided that certain menu items only appear if your process is Explorer.exe, a bit unfair for 3rd-party file explorers using IExplorerBrowser legitimately.

    • Paulo Pinto

      Specially since 3rd-party file explorers are the only usable way to write extensions in .NET and WinRT, as the shell keeps stuck in good old COM.

  • Brian Boorman

    I realize it's not the topic of this article, but since you link to the article about file extensions.... It would be nice if Microsoft followed it's own advice.

    I'm looking at you MS Edge.

    I have to use Edge as web-based interface to an internal data management tool. Even though in Default Apps settings I have PDF set to a third party tool Edge will decide that it is the PDF viewer after doing a save-as...

    Read more
    • iain clarke

      Raymond is on record as having commented that many of the sins he blames on “Contoso” are actually committed by Microsoft. No, he isn’t say which is which… So no company is safe from his disapproval, not even his own! He’s like Darkwing Duck…

    • Dave Gzorple

      Gah, I remember that, a few years ago when Edge was first introduced it really, really, really wanted to be the one and only PDF viewer and kept taking over all PDF management even though it did a really lousy job at it. This got so bad that Adobe did a walkthrough in their software when you ran it showing you all the steps you needed to take to disable this hijacking. I...

      Read more
    • Michael Taylor

      Go to the Edge settings. Then search for `PDF documents`. If `Always download PDF files` is checked then uncheck it. By default if Edge downloads a PDF then it auto-opens it in Edge. See if this solves your problem.