New Recreate At GPUVA Feature
After months of effort from both our hardware partners and us, the D3D12 team is happy to announce the preview of the Recreate At GPUVA feature as part of its Agility SDK Downloads 1.716 release. This feature is mostly intended to be used by tooling developers, such as PIX, to simplify capturing and replaying of GPU workloads.
The Problem:
Resource allocation behaves much like malloc in C, there are no guarantees about what memory address is given to the caller. Within a single process this is usually fine, until the application starts recording these addresses into buffers and using them in various routines (*cough cough ExecuteIndirect*). During an ExecuteIndirect()
call, the GPU may read a GPUVA out of the argument buffer and set it as a root descriptor before performing GPU work such as a Draw()
or a Dispatch()
.
The problem being that capture/replay tools cannot naively capture/replay the GPU work because the argument buffer contains capture-time GPUVAs. To work around this problem, without Recreate At GPUVA, tools would have to “intercept” these buffers and basically do a find-all-capture-time-GPUVAs and replace them with replay-time GPUVAs.
This solution is complicated to maintain, and it comes with its own problems. For example, it is difficult and sometimes impossible to “intercept” the argument buffer at capture time without perturbing the application’s GPU work. It also introduces some amount of non-determinism that we’re sure developers have noticed on these tools.
The Solution: Recreate at GPUVA
In order to help smoothen the development of ecosystem tools, the runtime team worked hard with our hardware partners to enable the specifying of a GPUVA before resource creation. This feature is, at the moment, only applicable to buffers and heaps. To ensure there’s adequate opportunity to weed out any issues, we’re keeping this feature at the preview level at this moment. We’re investigating what it would take to recreate shader identifiers and shader tables as well, so please let PIX, your hardware vendors, and us know if you would find that useful!
Sample code
#include <d3d12x.h> // Check for device support before using feature D3D12_FEATURE_DATA_D3D12_OPTIONS20 options20 = {}; if (pDevice->CheckFeatureSupport(D3D12_FEATURE_D3D12_OPTIONS20, &options20, sizeof(D3D12_FEATURE_DATA_D3D12_OPTIONS20)) != S_OK) { //Driver is not high enough to support recreate at return; } if (options20.RecreateAtTier != D3D12_RECREATE_AT_TIER_1) { //Driver does not report support for recreate at return; } const CD3DX12_RESOURCE_DESC1 BufDesc = CD3DX12_RESOURCE_DESC1::Buffer(1024); // Describe a buffer of 1024 using d3dx12.h helper const CD3DX12_HEAP_PROPERTIES HeapProp(D3D12_HEAP_TYPE_DEFAULT); // Describe the heap using d3dx12.h helper ID3D12Resource2* resource; pDevice->CreateCommittedResource2( HeapProp, D3D12_HEAP_FLAG_NONE, &BufDesc, D3D12_RESOURCE_STATE_COMMON, nullptr, nullptr, IID_PPV_ARGS(&resource)); // Get the virtual address from pageable tools D3D12_GPU_VIRTUAL_ADDRESS_RANGE addressRange; ID3D12PageableTools* pPageableTools; resource->QueryInterface(&pPageableTools); // QI into a pageable tools pPageableTools->GetAllocation(&addressRange); // Retrieve the GPUVA
// Reserve the gpuva before device creation
// Get the device factory so that we can have access to Tools and create our own device, note needs developer mode
ID3D12DeviceFactory* pDeviceFactory;
D3D12GetInterface(CLSID_D3D12DeviceFactory, IID_PPV_ARGS(&pDeviceFactory));
ID3D12Tools1* pD3D12Tools;
pDeviceFactory->GetConfigurationInterface(CLSID_D3D12Tools, IID_PPV_ARGS(&pD3D12Tools));
// Reserve GPUVAs before device creation
pD3D12Tools->ReserveGPUVARangesAtCreate(
&addressRange
,
1)
); // Create a device with reserved gpuva
pDeviceFactory->CreateDevice(
nullptr,
D3D_FEATURE_LEVEL_11_0,
IID_PPV_ARGS(&pDevice)
); // Create a device tools to reserve gpuva
ID3D12DeviceTools* pDeviceTools; pDevice->QueryInterface(&pDeviceTools); ID3D12Resource2* recreatedResource; //Call SetNextAllocationAddress with the given GPUVA before resource creation pDeviceTools->SetNextAllocationAddress(addressRange.StartAddress); pDevice->CreateCommittedResource2( &HeapProp, D3D12_HEAP_FLAG_NONE, &BufDesc, D3D12_RESOURCE_STATE_COMMON, nullptr, nullptr, IID_PPV_ARGS(&recreatedResource));
Conclusion
Huge thank you to our hardware partners that have worked hard to make this work. Please head to here to take a look at the API/specs. As a reminder, this is only available in developer mode.
Words from our Partners:
Intel:
“GPU-driven rendering poses challenges for GPU tools because applications can encode GPU virtual addresses within GPU resources. The new ‘RecreateAt’ API simplifies the tracking required to capture D3D12 streams from modern games and applications by eliminating the need to translate GPU virtual address ranges during playback. This API will enhance the accuracy, robustness, and performance of GPU tools moving forward.”
Head over to Intel’s website to download their drivers. Support for Recreate At GPUVA starts with driver 32.0.101.6557.
Nvidia:
“NVIDIA will fully support this SDK release, please contact your developer relations representative for specifics.”
AMD:
“AMD support for Recreate At GPUVA will be available in a developer preview driver in early February. A link to the driver will be posted here when it is available”
Qualcomm:
“Feature support is planned and to be available at a future date.”
0 comments
Be the first to start the discussion.