Is DEP on or off on Windows XP Service Pack 2?

Raymond Chen

Last time, we traced an IP_ON_HEAP failure to a shell extension that used an older version of ATL which was not DEP-friendly. But that led to a follow-up question:

Why aren’t we seeing this same crash in the main program as in the shell extension? That program uses the same version of ATL, but it doesn’t crash.

The reason is given in this chart. Notice that the default configuration is OptIn, which means that DEP is off for all processes by default, but is on for all Windows system components. That same part of the page describes how you can change to OptOut so that the default is to turn on DEP for all processes except for the ones you put on the exception list. There’s more information on this excerpt from the “Changes to Functionality in Microsoft Windows XP Service Pack 2” document. The program that comes with the shell extension is not part of Windows, so DEP is disabled by default. But Explorer is part of Windows, so DEP is enabled for Explorer by default. That’s why only Explorer encounters this problem. (This little saga does illustrate the double-edged sword of extensibility. If you make your system extensible, you allow other people to add features to it. On the other hand, you also allow other people to add bugs to it.)

The saga of the DEP exception is not over, however, because it turns out I’ve been lying to you. More information tomorrow.

Raymond Chen

Follow