Using automatic shader PDB resolution in PIX

Many features of PIX on Windows require access to shader debug information, but it can be a challenge to handle this efficiently. Shader debug information takes up a lot of space and isn’t necessary for executing shaders, so this data is often removed from the compiled shaders but doing so severely limits the help PIX on Windows can provide.

The Direct3D shader compilers allow you to strip debug data out of compiled shaders and place the debug data in a convenient location, typically outside of the disk image of your application. By following the guidelines below, you can set up your build system to produce debug data in a way that lets PIX discover and load shader symbols automatically. This allows you to create lean builds of your shaders without sacrificing the ability to inspect and debug your shaders.

There are a few steps to go through to enable this. The general process is outlined here, followed by best practices and then specific steps for DXBC and DXIL based shaders.

First, the compiler must emit debug data. Your build system must then extract the debug information and store it in a file. The compiler can offer a suggested name for the external debug data. This name will also be available to PIX, and is the key by which PIX can automatically load the debug data.

The suggested name is in fact a hash of the shader binary and, optionally, the HLSL source code that contributed to it. You can pass flags to the Direct3D compilers to control whether or not source is included in the hash.

Lastly, inform PIX of the location(s) of these saved PDB files via the Symbol / PDB Options section of the Settings menu.

Whenever shaders are examined or shader debugging is launched, the debug data should be automatically located and loaded. Any failures in this process will be reported when accessing the shader in the Pipeline or Shader Debugger view.

Note that the name provided by the compiler can be replaced, but this isn’t recommended since PIX will prefer to search for PDBs named with the shader’s hash. If you do supply your own name, ensure that it is unique, or PIX may load the wrong debug data file for DXBC shaders, since DXBC doesn’t embed a unique hash whereby PIX can be certain it has the correct PDB. User-supplied files can have embedded subdirectories, and PIX will attempt to find the debug data file in each subdirectory. For example, if you supply Symbols\Material137\PixelShader.PDB, PIX will search for that file appended to each of its PDB search paths, and then will try to find Material137\PixelShader.PDB, and finally PixelShader.PDB. Absolute paths are also supported. The string encoding is expected to be UTF-8.

PIX can also read PDBs from within .zip archive files.

PDB Best Practices

Shader build systems may produce 10k+ or even 100k+ of permutations and thus an enormous number of shader PDBs. A few best practices can drastically improve PIX performance when locating the necessary PDBs for a GPU Capture.

  1. Name your PDBs with their shader hash.

    When compiling your shaders, you can save the PDB into a .pdb file using either a custom specified file name or using the shader’s hash. It is recommended that you always use the shader’s hash as this will allow PIX a much easier time when locating PDBs later. This is easy to accomplish with for DXIL shaders via dxc:

     :: Instead of specifying the shader PDB file name like:
 dxc.exe myshader.hlsl /Zi /Fd C:\path\to\pdbs\myshader.pdb
 :: Just specify a directory to save the PDBs:
 dxc.exe myshader.hlsl /Zi /Fd C:\path\to\pdbs\
 :: And the shader hash will automatically be used for the filename

    Note the trailing \ on the directory.

  2. Consider using .zip archives to organize large quantities of PDB files, especially over 10k.

    For quantities over 10k PDBs, organizing and enumerating the numerous PDB files can be costly. PIX supports using .zip archives to store large collections of PDBs and speed up searching. To get the best results, it is recommended that your PDB .zip archives only contain PDB files (do not zip your entire build output including other non .pdb files) and to always follow recommendation number 1 about file naming using the shader hash.

  3. For extremely large quantities of PDB files exceeding 100k, consider organizing your PDB files into multiple .zip archives based upon build frequency.

    PIX can save time and efficiently skip enumerating .zip archives which it has previously scanned if the .zip archive is unchanged. For this reason, with extremely large build systems it is recommended that you organize your shader PDB files into multiple .zip archives, each containing PDBs that share similar build frequency. That way if you rebuild a subset of your shaders, many of the unchanged shaders will not need to update their PDB .zip archives.

    As a starting point recommendation, aim for .zip archives each containing about 5k-50k PDBs (this is not a hard rule, just a good starting point). So, a game with 250,000 shader permutations may want to split their PDBs and organize them into a few dozen .zip archives based upon the game’s levels, materials, or graphics subsystem. Then simply point PIX at the directory with all of the resulting .zip archives.

    For this quantity of PDBs it is extremely important that you follow recommendation number 1 about file naming using the shader hash for the best performance.

Slim PDBs (DXIL only)

A Slim PDB is generated using /Zs when compiling a shader. The resulting PDB contains only the source files and compilation args and does not have the full debug info, but is significantly smaller in size than a full PDB. With a Slim PDB, PIX can display the shader source and compilation args, but shader debugging is disabled. To enable shader debugging, , you can generate a full PDB by opening the Commands panel from the toolbar at the top of the view.

Note: If a different version of the compiler is used to generate the full PDB (PIX ships with the latest compiler), different IL and GPU instructions can be produced as a result and PIX will notify you of the difference. To use a specific compiler, you can replace the dxcompiler.dll located next to WinPixe.exe, and then re-open the capture.

Emitting PDBs for DXIL Shaders (compiled via dxcompiler.dll)

You can discover the relevant functions for enabling this for DXIL shaders via a factory function exposed by dxcompiler.dll as illustrated below.

First, compile your shader with the following flags:

  • /Zi – include debug information.
  • /Zss or /Zsb – generate a suggested name for the debug data. Note that /Zss means that both the HLSL and shader binary will be hashed to generate the name. /Zsb means that the source will not be considered.

The DXIL compiler offers a one-step mechanism for compiling and extracting debug information. Use the IDxcCompiler3::Compile method, defined in dxcapi.h. The IDxcResult interface allows you to retrieve a pointer to the suggested debug data file name, and the debug data itself. Simply write the data to a file with the suggested name.

If it is inconvenient to use the one-step method, the following manual process can be followed to remove debug data and name the LLD files. Note that error handling has been omitted for clarity.

#include "DxilContainer.h"
#include "DXCApi.h"

Using namespace hlsl;

// First create the factory interface through which other Dxc interfaces can be retrieved:
decltype(&DxcCreateInstance) pfnDxcCreateInstance = 
    (decltype(&DxcCreateInstance))GetProcAddress(dxcompiler_dll, "DxcCreateInstance");

// Build a reflector for the existing container
ComPtr<IDxcContainerReflection> pDxcContainerReflection;
(*pfnDxcCreateInstance)(CLSID_DxcContainerReflection, IID_PPV_ARGS(pDxcContainerReflection.GetAddressOf()));

// Use this or any other mechanism for creating a simple COM object that publishes an IDxcBlob
// that wraps your shader binary.
ComPtr<IDxcBlob> pContainer = QuickCom::Make<DxcBlob>(pShaderBytecode, BytecodeLength);

// Load your shader binary into the container reflector:
pDxcContainerReflection->Load(pContainer.Get());

// Find which part index contains the shader name and retrieve it:
UINT32 debugNameIndex;
pDxcContainerReflection->FindFirstPartKind(DFCC_ShaderDebugName, &debugNameIndex);
ComPtr<IDxcBlob> pPDBName;
pDxcContainerReflection->GetPartContent(debugNameIndex, pPDBName.GetAddressOf());

// Find which part index contains the debug data and retrieve it:
UINT32 debugPartIndex;
pDxcContainerReflection->FindFirstPartKind(DFCC_ShaderDebugInfoDXIL, &debugPartIndex);
ComPtr<IDxcBlob> pPDB;
pDxcContainerReflection->GetPartContent(debugPartIndex, pPDB.GetAddressOf());

// DxilShaderDebugName is defined in DxilContainer.h
pDebugNameData = reinterpret_cast<const DxilShaderDebugName *>(pPDBName->GetBufferPointer());
auto pName = reinterpret_cast<const char *>( pDebugNameData + 1);

// Now write the contents of pPDB to a file named by pName
// Not illustrated here

// Lastly, create a new shader container with the debug data removed
ComPtr<IDxcContainerBuilder> pDxcContainerBuilder;
(*pfnDxcCreateInstance)(CLSID_DxcContainerBuilder, IID_PPV_ARGS(pDxcContainerBuilder.GetAddressOf()));
pDxcContainerBuilder->Load(pContainer.Get());
pDxcContainerBuilder->RemovePart(DFCC_ShaderDebugInfoDXIL);
ComPtr<IDxcOperationResult> pStrippedResult;
pDxcContainerBuilder->SerializeContainer(pStrippedResult.GetAddressOf());
pStrippedResult->GetResult(pStripped.GetAddressOf());

// Use the contents of pStripped as your final shader

Emitting PDBs for DXBC shaders (compiled with fxc or via d3dcompiler*.dll)

Fxc.exe offers command-line support for generating correctly-named PDBs directly. You need FXC from the Windows 10 Fall Creator’s Update SDK (version 16299) or later. Use the command line below:

fxc.exe /Zi /Zss /Fd .\

Those parameters are: /Zi Turn on debug information. /Zss Include source when generating hash for the PDB name (the alternative is /Zsb, which doesn’t include source code in the hash). /Fd <outputdirectory>\ Name an output directory instead of a PDB name.

Note the trailing backslash on the argument for /Fd. This trailing backslash tells FXC to generate the hash-names for the PDB and place the resulting files in the specified directory.

If fxc.exe is not part of your tool chain, several APIs are defined in d3dcompiler.h that allow you to extract debug data from a compiled shader and place it into a file in a location convenient to your situation. These APIs are flat DLL exports from d3dcompiler*.dll.

First of all, compile your shaders with the following flags:

  • D3DCOMPILE_DEBUG: This flag causes the compiler to emit debug information into the output container.
  • D3DCOMPILE_DEBUG_NAME_FOR_SOURCE or D3DCOMPILE_DEBUG_NAME_FOR_BINARY: These flags cause the compiler to emit the suggested name for the debug data file. Note that FOR_SOURCE means that the suggested name will be a hash of both the HLSL source code and the resulting object code. FOR_BINARY means that only the binary will be considered.

To remove the debug data from the shader and name the PDB files do the following. Note that error handling has been omitted for clarity.

// Starting with a compiled shader in pShaderBytecode of length BytecodeLength,
// retrieve the debug info part of the shader:
ComPtr<ID3DBlob> pPDB;
D3DGetBlobPart(pShaderBytecode, BytecodeLength, D3D_BLOB_PDB, 0, pPDB.GetAddressOf());

// Now retrieve the suggested name for the debug data file:
ComPtr<ID3DBlob> pPDBName;
D3DGetBlobPart(pShaderBytecode, BytecodeLength, D3D_BLOB_DEBUG_NAME, 0, pPDBName.GetAddressOf());

// This struct represents the first four bytes of the name blob:
struct ShaderDebugName 
{
    uint16_t Flags;       // Reserved, must be set to zero.
    uint16_t NameLength;  // Length of the debug name, without null terminator.
                          // Followed by NameLength bytes of the UTF-8-encoded name.
                          // Followed by a null terminator.
                          // Followed by [0-3] zero bytes to align to a 4-byte boundary.
};

auto pDebugNameData = reinterpret_cast<const ShaderDebugName *>(pPDBName->GetBufferPointer());
auto pName = reinterpret_cast<const char *>( pDebugNameData + 1);

// Now write the contents of the blob pPDB to a file named the value of pName
// Not illustrated here

// Now remove the debug info from the target shader, resulting in a smaller shader
// in your final application’s data:
ComPtr<ID3DBlob> pStripped;

D3DStripShader(pShaderBytecode, BytecodeLength, D3DCOMPILER_STRIP_DEBUG_INFO, pStripped.GetAddressOf());

// Finally, write the contents of pStripped as your final shader file.

Supplying a Custom Debug File Name

The /Fd argument for dxc is the recommended method of naming the debug file, since this will result in the correct shader hash being recorded in the shader, and the correct matching name being suggested for the pdb, allowing PIX to connect the two.

If that method isn’t available to you, follow this process to supply your own debug file name in such a way that PIX can discover it in your application’s shader binaries. Error handling has been removed for clarity:

// Any valid UTF-8 path is allowed
const char pNewName[] = "MyOwnUniqueName.lld";

// Blobs are always a multiple of 4 bytes long. Since DxilShaderDebugName
// is itself 4 bytes, we pad the storage of the string (not the string itself)
// to 4 bytes also.
size_t lengthOfNameStorage = (_countof(pNewName) + 0x3) & ~0x3;

// See above for the definition of DxilShaderDebugName
size_t nameBlobPartSize = sizeof(DxilShaderDebugName) + lengthOfNameStorage;

auto pNameBlobContent = reinterpret_cast<DxilShaderDebugName*>(malloc(nameBlobPartSize));

// Ensure bytes after name are indeed zeroes:
ZeroMemory(pNameBlobContent, nameBlobPartSize);

pNameBlobContent->Flags = 0;
// declared length does not include the null terminator:
pNameBlobContent->NameLength = _countof(pNewName) - 1;
// but the null terminator is expected to be present:
memcpy(pNameBlobContent + 1, pNewName, _countof(pNewName));

ComPtr<ID3DBlob> pShaderWithNewName;
D3DSetBlobPart(pShaderBytecode, BytecodeLength, D3D_BLOB_DEBUG_NAME, 0, pNameBlobContent, 
               nameBlobPartSize, pShaderWithNewName.GetAddressOf());

// D3DSetBlobPart returns a complete new blob container:
pShaderBytecode = pShaderWithNewName->GetBufferPointer();
BytecodeLength = pShaderWithNewName->GetBufferSize();

free(pNameBlobContent);