Adventures in application compatibility: Calling an internal function

Raymond Chen

Raymond

We try hard to make sure applications continue to work, but some things that applications do are so egregious that there’s no practical way of getting them to work.

Today, we’ll learn about one such.

The program bills itself as “the most advanced Windows optimization toolkit in the universe!”

If you say so.

One of their awesome optimizations, it appears, is to reset file associations to match their concept of what file associations should be in an ideal world. This ideal world probably is one in which their application is the default handler for a lot of popular and contentious file types.

The application compatibility team reported that this program crashed when you asked it to reset file associations. Windows goes to some lengths to make it hard for programs to change file associations programmatically, and instead of trying to reverse-engineer how Windows protects the settings in the registry, they instead opted to reverse-engineer the code that manages the settings.

Specifically, they scanned memory looking for the internal function that sets the file associations, and then called it.

Now, searching all of memory is a daunting task, but they were able to take a shortcut: They got their hands on an IApplication¬≠Association¬≠Registration object, which is the documented interface for managing application defaults. They used the vtable as a clue as to where the application defaults management code is, and focused their search on that region of memory. I’m not quite sure exactly how they found the internal function; perhaps they disassembled the code looking for call instructions, and assumed that the third call (say) was to a handy function, and then they disassembled the handy function and assumed that the second call (say) was to the secret internal function.

Of course, searching memory for a function to call is not exactly something documented and supported. Windows made some changes to how these functions operate, and that threw off their code that grovels the binary, and they ended up calling the wrong function.

Instead of creating a decoy that keeps their crazy algorithm working, the team opted to let the program crash when you pushed the button to reset file associations to their ideal state. This was an older version of a program still under active development, and the failure mode made it rather clear to the user that the program was at fault: It crashes when you press a specific button. The initial inclination is to blame that button. Therefore, the user will contact the vendor for an update.

Now that everything is online, shifting the cost of a vendor’s mistake to the vendor’s support infrastructure has become a viable alternative to patching the operating system to work around a single program.

 

Raymond Chen
Raymond Chen

Follow Raymond   

9 comments

Comments are closed.

  • Avatar
    Kirill Illenseer

    After a couple times I screwed myself over with “optimization” tools (hey, we all were gullible when looking back long enough) and a countless couple times more I’ve been cleaning up trash on family & friends’ computers, I dare to say than an “optimization” tool not working anymore is in fact an advancement, something to celebrate. Even more so if it’s “advanced”.

  • Avatar
    Ionel POP

    I’m wondering how do you call such an internal function? Just load the DLL and jump to an offset? Or they make the memory search each time they need to call it?
    Anyway, it seems pretty fragile as solution. I’m surprised they released in production.

    • Avatar
      kadu cortez

      Believe me, its never a pretty solution but sometimes it’s surprisingly useful. Once i worked with a framework for interface components in Delphi and the only solution i could work out for a random crash was a patch on the fly to call an internal initialization function and then overwrite the first bytes of the function to write a ret to disable further calls. Over 5 senior developers tried to read the actual code to understand and fix the issue, but the code is such trash that my hack had been there for over 6 years now and nobody has a clue on why it crashes.

  • Avatar
    Me Gusta

    I really dislike writing about internals. The last time I did it, I basically disproved every assumption someone made on the behaviour of the VirtualAlloc family of functions when you don’t use a base address.
    I tend to hate doing this the other way around because people then seem to take this as a sign that someone is giving their approval and somehow means that Microsoft will never change this behaviour.

  • Avatar
    Keith Patrick

    As bad a practice as that is, I can definitely see the allure of it – it’s just clever enough and into system details to where you can almost feel like you’re outsmarting the Windows team, even though you’re outsmarting yourself in the long run.  I’ve fallen into the temptation in .Net by using GetType().GetMethod(BindingFlags.NonPublic|…) a couple of times as *temporary* workarounds, but at least in those cases, the technique is mundane enough to where I don’t get wedded to the idea

  • Avatar
    Piotr Siódmak

    Oh the times you spied on windows messages to write a program that automates another that doesn’t have a public API (I’m looking at you, [popular communication program])