Shell extensions that create worker threads need to call the SHGetInstanceExplorer function so that Explorer will not exit while the worker thread is still running. When your worker thread finishes, you release the IUnknown that you obtained to tell the host program, “Okay, I’m done now, thanks for waiting.”

You can read this contract from the other side. Instead of thinking of yourself as the shell extension running inside a host program, think of yourself as the host program that has a shell extension running inside of it. Consider a simple program that displays the properties of a file, or at least tries to:

#include <windows.h>
#include <shellapi.h>
#include <tchar.h>
int __cdecl _tmain(int argc, TCHAR **argv)
{
  SHELLEXECUTEINFO sei = { sizeof(sei) };
  sei.fMask = SEE_MASK_FLAG_DDEWAIT | SEE_MASK_INVOKEIDLIST;
  sei.nShow = SW_SHOWNORMAL;
  sei.lpVerb = TEXT("properties");
  sei.lpFile = TEXT("C:\\Windows\\Explorer.exe");
  ShellExecuteEx(&sei);
  return 0;
}

Oh dear! When you run this program, nothing happens. Well, actually, something did happen, but the program exited too fast for you to see it. To slow things down, add the line

  MessageBox(NULL, TEXT("Waiting"), TEXT("Title"), MB_OK);

right before the return 0. Run the program again, and this time the properties dialog appears, as well as the message box. Aha, the problem is that our program is exiting while the property sheet is still active. (Now delete that call to MessageBox before something stupid happens.)

The question now is how to know when the property sheet is finished so we can exit. That’s where SHSetInstanceExplorer comes in. The name “Explorer” in the function name is really a placeholder for “the host program”; it just happens to be called “Explorer” because the function was written from the point of view of the shell extension, and the host program is nearly always Explorer.exe.

In this case, however, we are the host program, not Explorer. The SHSetInstanceExplorer lets you register a free-threaded IUnknown that shell extensions can obtain by calling SHGetInstanceExplorer. Following COM reference counting conventions, the SHGetInstanceExplorer performs an AddRef() on the IUnknown that it returns; the shell extension’s worker thread performs the corresponding Release() when it is finished.

All that is required of the IUnknown that you pass to SHSetInstanceExplorer is that it be free-threaded; in other words, that it support being called from multiple threads. This means managing the “process reference count” with interlocked functions rather than boring ++ and -- operators. Of course, in practice, you also need to tell your main program “Okay, all the shell extensions are finished; you can exit now” when the reference count drops to zero.

There are many ways to accomplish this task. Here’s one that I threw together just now. I didn’t think too hard about this class; I’m not positioning this as the best way of implementing it, or even as a particularly good one. The purpose of this article is to show the principle behind process references. Once you understand that, you are free to go ahead and solve the problem your own way. But here’s a way.

#include <shlobj.h>
class ProcessReference : public IUnknown {
public:
  STDMETHODIMP QueryInterface(REFIID riid, void **ppv)
  {
    if (riid == IID_IUnknown) {
      *ppv = static_cast<IUnknown*>(this);
      AddRef();
      return S_OK;
    }
   *ppv = NULL; return E_NOINTERFACE;
  }
  STDMETHODIMP_(ULONG) AddRef()
    { return InterlockedIncrement(&m_cRef); }
  STDMETHODIMP_(ULONG) Release()
  {
    LONG lRef = InterlockedDecrement(&m_cRef);
    if (lRef == 0) PostThreadMessage(m_dwThread, WM_NULL, 0, 0);
    return lRef;
  }
  ProcessReference()
    : m_cRef(1), m_dwThread(GetCurrentThreadId())
    { SHSetInstanceExplorer(this); }
  ~ProcessReference()
  {
    SHSetInstanceExplorer(NULL);
    Release();
    MSG msg;
    while (m_cRef && GetMessage(&msg, NULL, 0, 0)) {
      TranslateMessage(&msg);
      DispatchMessage(&msg);
    }
  }
private:
  LONG m_cRef;
  DWORD m_dwThread;
};

The idea behind this class is that the main thread (and only the main thread) creates it on the stack. When constructed, the object registers itself as the “process IUnknown“; any shell extensions that call SHGetInstanceExplorer will get a pointer to this object. When the object is destructed, it unregisters itself as the process reference (to avoid dangling references) and waits for the reference count to drop to zero. Notice that the Release method posts a dummy thread message so that the “waiting for the reference count to go to zero” message loop will wake up.

In a sense, this is backwards from the way normal COM objects work, which operate under the principle of “When the reference count drops to zero, the object is destructed.” We turn it around and code it up as “when the object is destructed, it waits for the reference count to drop to zero.” If you wanted to do it the more traditional COM way, you could have the main thread go into a wait loop and have the object’s destructor signal the main thread. I did it this way because it makes using the class very convenient.

Now that we have a process reference object, it’s a simple matter of adding it to our main thread:

int __cdecl _tmain(int argc, TCHAR **argv)
{
  ProcessReference ref;
  ...

With this modification, the program displays the property sheet and patiently waits for the property sheet to be dismissed before it exits.

Exercise: Explain how the object behaves if we initialized the reference count to zero and deleted the call to Release in the destructor.

Bonus reading: Do you know when your destructors run? Part 2.