September 14th, 2023

I accidentally performed an operation on INVALID_HANDLE_VALUE, and it worked: What just happened?

Raymond Chen
Raymond Chen

Suppose you have some code that wants to open a file and share it with another process.

HANDLE file = CreateFile(...);

HANDLE dup;
if (DuplicateHandle(
        GetCurrentProcess(), /* source process */
        file, /* source handle */
        targetProcess,
        &dup,
        0,
        FALSE, /* no inherit */
        DUPLICATE_SAME_ACCESS)) {
    ... tell the other process to use "dup" ...
}

CloseHandle(file);

This code is missing some critical error handling: What if the CreateFile fails?

If CreateFile fails, it returns INVALID_HANDLE_VALUE. This code then passes INVALID_HANDLE_VALUE as the source handle to duplicate into the other process. But instead of failing, the Duplicate­Handle succeeds and produces a handle. What handle just got duplicated?

Some time ago, we studied why HANDLE return values are so inconsistent, and I mentioned that

By coincidence, the value INVALID_HANDLE_VALUE happens to be numerically equal to the pseudohandle returned by GetCurrentProcess().

Therefore, what happened is that we gave a copy of our own process handle to the other process.

Oops.

But wait, there’s more. The handle returned by GetCurrentProcess() has PROCESS_ALL_ACCESS permission. Not only did you give the other process the wrong thing, you gave it possibly the worst possible thing: You gave it full control over your own process.

Double oops.

Why does GetCurrentProcess() return a pseudo-handle value that matches a common error case that people could overlook? I don’t know, but I have an idea.

Developer 1: “Hey, what fake handle value should GetCurrentProcess() return?”

Developer 2: “I dunno. We need to pick something that is guaranteed never to accidentally match a real handle value.”

Developer 1: “Look, there’s this special value INVALID_HANDLE_VALUE that is returned to indicate that an error occurred. This is provably a handle value that can never match a real handle, since it is used to indicate a problem.”

Developer 2: “Great, let’s use that!”

It seemed like a good idea at the time. For a special value, use something that couldn’t possibly conflict with a normal value.

Unfortunately, people are fallible, and bugs can occur like

HANDLE file = INVALID_HANDLE_VALUE;

if (want_file) {
    file = CreateFile(...);
    if (file == INVALID_HANDLE_VALUE) {
        error();
        return;
    }

/* other intervening code */

// Okay, give the file to the other process.
HANDLE dup;
if (DuplicateHandle(
        GetCurrentProcess(), /* source process */
        file, /* source handle */
        targetProcess,
        &dup,
        0,
        FALSE, /* no inherit */
        DUPLICATE_SAME_ACCESS)) {
    ... tell the other process to use "dup" ...
}

CloseHandle(file);

The code that calls Duplicate­Handle forgot to check whether want_file is set and inadvertently passed INVALID_HANDLE_VALUE. Oops, duplicated a full-access process handle.

This pattern of “using an invalid value to carry a special alternate meaning” is actually quite common. For example, Set­Windows­Hook­Ex uses a thread ID of zero to mean “global hook”, since zero is not a valid thread ID. And many functions imbue the special value nullptr with all sorts of special meaning.

In retrospect, the choice to use INVALID_HANDLE_VALUE as a pseudohandle was unfortunate. But what happened happened. You can’t change the past, and in computer software, you have to live with your mistakes.

Bonus chatter: RPC has the system_handle attribute which lets you pass kernel handles via RPC, and you also have to say what kind of kernel handle you intend to pass. For files, you say system_handle(sh_file). If you pass the wrong kind of handle, the operation fails. There’s a sample in the Windows Classic Samples repo.

Category
Old New Thing
Topics
Code

Author

Raymond Chen

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

7 comments

Discussion is closed. Login to edit/delete existing comments.

Newest
Newest Popular Oldest
  • Christian Ullrich · Edited

    A system_handle can be to all kinds of files, processes, synchronization objects, all proper serious kernel stuff … or a “DirectComposition surface”. I’m sure there’s a story behind that one.

    • skSdnW

      FindFirstFile also returns a HANDLE that is not really a kernel handle. FindClose existing should be your first clue for that one.

  • Shawn Van Ness

    iirc it’s just -1, right .. was this done as a performance optimization — because it’s so common to need a handle to the current process?

    or were there some issues with process-shutdown, if a process holds open a (real) handle to itself?

    • Henke37

      Processes can easily have a real handle to themselves. Such as doing a DuplicateHandle to its own memory.

  • Henke37

    Another pseudo-handle value that you shouldn’t accidentally copy to another process is the one returned by GetCurrentThread. That’s -2 for those who are curious. And yes, it’s pretty much as powerful as the process handle, since it has THREAD_ALL_ACCESS on your current thread.

  • Richard Yu

    “The 16-bit functions OpenFile, _lopen and _lcreat return -1 on failure”
    Why not returning zero?
    Did you notice that “_lopen” and “_lcreat” looks like POSIX’s open and creat?
    So the answer is: because POSIX returns -1.

    • Yuri Khan

      And if you want to dig deeper, POSIX could not return zero because zero is the standard input file descriptor. (Standard output and error are 1 and 2 respectively.)

Read next