Adventures in application compatibility: Calling an internal function
Raymond Chen
We try hard to make sure applications continue to work, but some things that applications do are so egregious that there’s no practical way of getting them to work.
Today, we’ll learn about one such.
The program bills itself as “the most advanced Windows optimization toolkit in the universe!”
If you say so.
One of their awesome optimizations, it appears, is to reset file associations to match their concept of what file associations should be in an ideal world. This ideal world probably is one in which their application is the default handler for a lot of popular and contentious file types.
The application compatibility team reported that this program crashed when you asked it to reset file associations. Windows goes to some lengths to make it hard for programs to change file associations programmatically, and instead of trying to reverse-engineer how Windows protects the settings in the registry, they instead opted to reverse-engineer the code that manages the settings.
Specifically, they scanned memory looking for the internal function that sets the file associations, and then called it.
Now, searching all of memory is a daunting task, but they were able to take a shortcut: They got their hands on an IApplicationÂAssociationÂRegistration object, which is the documented interface for managing application defaults. They used the vtable as a clue as to where the application defaults management code is, and focused their search on that region of memory. I’m not quite sure exactly how they found the internal function; perhaps they disassembled the code looking for call instructions, and assumed that the third call (say) was to a handy function, and then they disassembled the handy function and assumed that the second call (say) was to the secret internal function.
Of course, searching memory for a function to call is not exactly something documented and supported. Windows made some changes to how these functions operate, and that threw off their code that grovels the binary, and they ended up calling the wrong function.
Instead of creating a decoy that keeps their crazy algorithm working, the team opted to let the program crash when you pushed the button to reset file associations to their ideal state. This was an older version of a program still under active development, and the failure mode made it rather clear to the user that the program was at fault: It crashes when you press a specific button. The initial inclination is to blame that button. Therefore, the user will contact the vendor for an update.
Now that everything is online, shifting the cost of a vendor’s mistake to the vendor’s support infrastructure has become a viable alternative to patching the operating system to work around a single program.
Light
Dark
9 comments
After a couple times I screwed myself over with “optimization” tools (hey, we all were gullible when looking back long enough) and a countless couple times more I’ve been cleaning up trash on family & friends’ computers, I dare to say than an “optimization” tool not working anymore is in fact an advancement, something to celebrate. Even more so if it’s “advanced”.
> trying to reverse-engineer how Windows protects the settings in the registry
“Choose defaults by file type” -> .gz -> “Choose a default” -> “Look for a program in the microsoft store”
Dude let me type a command line here. (I really want to type in c:\cygwin64\bin\gzip.exe -k)
Dude let me change the icon of a file.
I’m wondering how do you call such an internal function? Just load the DLL and jump to an offset? Or they make the memory search each time they need to call it?
Anyway, it seems pretty fragile as solution. I’m surprised they released in production.
Believe me, its never a pretty solution but sometimes it’s surprisingly useful. Once i worked with a framework for interface components in Delphi and the only solution i could work out for a random crash was a patch on the fly to call an internal initialization function and then overwrite the first bytes of the function to write a ret to disable further calls. Over 5 senior developers tried to read the actual code to understand and fix the issue, but the code is such trash that my hack had been there for over 6 years now and nobody has a clue on why it crashes.
I really dislike writing about internals. The last time I did it, I basically disproved every assumption someone made on the behaviour of the VirtualAlloc family of functions when you don’t use a base address.
I tend to hate doing this the other way around because people then seem to take this as a sign that someone is giving their approval and somehow means that Microsoft will never change this behaviour.
As bad a practice as that is, I can definitely see the allure of it – it’s just clever enough and into system details to where you can almost feel like you’re outsmarting the Windows team, even though you’re outsmarting yourself in the long run. I’ve fallen into the temptation in .Net by using GetType().GetMethod(BindingFlags.NonPublic|…) a couple of times as *temporary* workarounds, but at least in those cases, the technique is mundane enough to where I don’t get wedded to the idea
Oh the times you spied on windows messages to write a program that automates another that doesn’t have a public API (I’m looking at you, [popular communication program])
I bet the team was rather happy to let this program keep crashing.
But you have to admit: that was pretty impressive.