Adventures in application compatibility: The case of the RAII type that failed to run its destructor

Raymond Chen

A customer reported that their system crashed with bugcheck code 0xEF: CRITICAL_PROCESS_DIED because the RPCSS service died. The investigation of this failure was difficult because by the time the crash occurred, everything was badly corrupted with no evidence as to who did the corrupting. The data that was being corrupted is per-thread data that is normally managed by an RAII type, so there should be no way it could be getting corrupted, assuming that the RAII type is being used properly: Always created and destructed in LIFO order, which is typically accomplished by making it a local variable in a non-coroutine function.

The team scoured their code for all uses of that RAII type and couldn’t find any place where they could possibly be using it incorrectly.

Eventually, the team discovered that a third party anti-malware application had injected itself into the RPCSS service and was detouring some functions.¹ And for some reason, the detour was unhappy, but instead of failing the call, it raised a structured exception.

This structured exception was caught by the RPC equivalent of the giant try/catch that COM wraps around every server method. For RPC, this wrapper goes by the name Rpc­Exception­Filter. The exception was raised from their detour and it so happens that the exception they raised is not one that the Rpc­Exception­Filter function deems to be noteworthy, so the Rpc­Exception­Filter function swallows the exception and returns an RPC failure to the client.

As I noted, this structured exception bypasses C++ destructors.

And that’s why the RAII class was not working. The code wasn’t holding it wrong. Rather, the third party code broke the rules.

Now, the third party code had this problem all along: The rogue exception caused an entire RPC call to become abandoned mid-stream, resulting in lots of leaks and missing cleanup. What changed is that there’s new code in the RPCSS service that was counting on the cleanup to avoid a crash.

This particular anti-malware program must be somewhat popular because the problem recurs about once or twice a year with a different customer each time. This is the bitter spot² in bug frequency, because it’s not frequent enough that the problem remains on your mind whenever you get a mystery crash so you can open the discussion with the customer by saying, “Okay, before I spend too much time on this, can you tell me if you recently installed or reconfigured Contoso anti-malware?” On the other hand, the issue is frequent enough that it consumes a lot of time and effort to gather crash dumps and eventually realize that it’s that Contoso anti-malware problem again.

¹ Windows does not support detouring the operating system. This anti-malware software is doing unsupported things, but good luck explaining that to their customers.

² The bitter spot is the opposite of the sweet spot. Or is the opposite of the sweet spot the salty spot? The sour spot? Maybe it’s all of them. When you find yourself in this spot, you are probably going to be sour, bitter and probably a little salty.


Discussion is closed. Login to edit/delete existing comments.

  • Henry Skoglund 0

    When helping someone with a weirdly behaving program, in the old days #1 advice used to be “Upgrade your video driver” but today I always start with “Do you have any 3rd-party antivirus program installed” because sadly they are often the culprit.
    Also thank you for making me laugh: “… The code wasn’t holding it wrong. . “” 🙂

  • 紅樓鍮 0

    Bonus chatter: Rust considers RAII leaks (which Rust supports officially through e. g. std::mem::forget) to be safe. This is normally OK, until someone tries to rely on destructors being run to guarantee safety, like std::thread::scoped did ( thread-scoped).

    Rust reconciled them with a new continuation-passing style API (std::thread::scope), which by design prevents forgetting the join guard.
    (Not that it would help the case of an unhandled structured exception.)

    Bonus bonus chatter: Doing anything that involves LoadLibrary‘ing is considered inherently unsafe in Rust, since you can’t guarantee the contents of the DLL. And what’s more fun, in many scenarios there’s just no Rust-safe alternatives to doing LoadLibrary. Meaning there exist applications for which it’s impossible to be Rust-safe, for legitimate reasons.

    I think this makes me rethink the whole thing about “program correctness”: the concept is just difficult to apply once you go out of the language abstract machine level. If you have the privileges, you can do everything to drive a proven-correct Lean program nuts, at every level from patching the OS loader to injecting rogue threads. Hey, even Rust is using LLVM for codegen, which is not proven-correct (and is known to have bugs).

  • Jacob Manaker 0

    Eventually, the team discovered that a third party anti-malware application had injected itself into the RPCSS service and was detouring some functions.

    The first time I read this, I missed the “anti-” in “anti-malware application.” It made a lot more sense that way.

    • Nathan Williams 0

      Anti-malware software is the most common type of malware after all.

  • David Wolff 0

    When I worked at Prime Computer, we had an automated dump-analysis tool that had rules about what things in a crash dump were clues to known crashes. For example, if a particular routine crashed, and the third parameter was invalid, it was a known crashing bug (fixed in a later release). Doesn’t Microsoft have something similar?

  • Alois Kraus 0

    It could help to get a short wpr recording of the machine which collects all loaded modules besides a memory dump as default action. I had my own share of AV induced issues and did write to be able to quickly query all known AV vendors in the stacks. With proper stacktags one can quickly find out if the slowness of an application comes from SmartScreen, Mc Affee or any other AV vendor. I plan to blog more about known patterns of performance degradations caused by AV engines which are with other tools notoriously hard to find.
    I assume that you have an automatic analyzer for memory dumps which scans for known problematic modules which should not be loaded in central system processes.

  • Henke37 0

    Fighting games seem to have settled on “sour spot”.

  • Richard Russell 0

    You left out the umami spot.

Feedback usabilla icon