Adventures in application compatibility: Calling an internal function

Raymond Chen


We try hard to make sure applications continue to work, but some things that applications do are so egregious that there’s no practical way of getting them to work.

Today, we’ll learn about one such.

The program bills itself as “the most advanced Windows optimization toolkit in the universe!”

If you say so.

One of their awesome optimizations, it appears, is to reset file associations to match their concept of what file associations should be in an ideal world. This ideal world probably is one in which their application is the default handler for a lot of popular and contentious file types.

The application compatibility team reported that this program crashed when you asked it to reset file associations. Windows goes to some lengths to make it hard for programs to change file associations programmatically, and instead of trying to reverse-engineer how Windows protects the settings in the registry, they instead opted to reverse-engineer the code that manages the settings.

Specifically, they scanned memory looking for the internal function that sets the file associations, and then called it.

Now, searching all of memory is a daunting task, but they were able to take a shortcut: They got their hands on an IApplication¬≠Association¬≠Registration object, which is the documented interface for managing application defaults. They used the vtable as a clue as to where the application defaults management code is, and focused their search on that region of memory. I’m not quite sure exactly how they found the internal function; perhaps they disassembled the code looking for call instructions, and assumed that the third call (say) was to a handy function, and then they disassembled the handy function and assumed that the second call (say) was to the secret internal function.

Of course, searching memory for a function to call is not exactly something documented and supported. Windows made some changes to how these functions operate, and that threw off their code that grovels the binary, and they ended up calling the wrong function.

Instead of creating a decoy that keeps their crazy algorithm working, the team opted to let the program crash when you pushed the button to reset file associations to their ideal state. This was an older version of a program still under active development, and the failure mode made it rather clear to the user that the program was at fault: It crashes when you press a specific button. The initial inclination is to blame that button. Therefore, the user will contact the vendor for an update.

Now that everything is online, shifting the cost of a vendor’s mistake to the vendor’s support infrastructure has become a viable alternative to patching the operating system to work around a single program.


Raymond Chen
Raymond Chen

Follow Raymond   

Kirill Illenseer 2019-07-24 07:47:15
After a couple times I screwed myself over with "optimization" tools (hey, we all were gullible when looking back long enough) and a countless couple times more I've been cleaning up trash on family & friends' computers, I dare to say than an "optimization" tool not working anymore is in fact an advancement, something to celebrate. Even more so if it's "advanced".
Joshua Hudson 2019-07-24 08:25:03
> trying to reverse-engineer how Windows protects the settings in the registry "Choose defaults by file type" -> .gz -> "Choose a default" -> "Look for a program in the microsoft store" Dude let me type a command line here. (I really want to type in c:\cygwin64\bin\gzip.exe -k) Dude let me change the icon of a file.
Ionel POP 2019-07-24 08:30:03
I'm wondering how do you call such an internal function? Just load the DLL and jump to an offset? Or they make the memory search each time they need to call it? Anyway, it seems pretty fragile as solution. I’m surprised they released in production.
Me Gusta 2019-07-24 08:44:06
I really dislike writing about internals. The last time I did it, I basically disproved every assumption someone made on the behaviour of the VirtualAlloc family of functions when you don't use a base address. I tend to hate doing this the other way around because people then seem to take this as a sign that someone is giving their approval and somehow means that Microsoft will never change this behaviour.
Keith Patrick 2019-07-24 11:18:06
As bad a practice as that is, I can definitely see the allure of it - it's just clever enough and into system details to where you can almost feel like you're outsmarting the Windows team, even though you're outsmarting yourself in the long run.  I've fallen into the temptation in .Net by using GetType().GetMethod(BindingFlags.NonPublic|...) a couple of times as *temporary* workarounds, but at least in those cases, the technique is mundane enough to where I don't get wedded to the idea
Piotr Siódmak 2019-07-24 12:10:56
Oh the times you spied on windows messages to write a program that automates another that doesn't have a public API (I'm looking at you, [popular communication program])
Marc K 2019-07-24 13:26:36
I bet the team was rather happy to let this program keep crashing.
Ian Boyd 2019-07-25 19:06:32
But you have to admit: that was pretty impressive.