Fix multi-GPU inference on Blackwell (dual RTX 5080): four stacked fixes
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin, zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has a Myelin l2cm crash when creating a context on a second Blackwell device). 1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1 memory with a device-0 stream, causing an async illegal memory access that poisoned every CUDA context (sticky 700). Track owning device and rebuild the cache on device switch. 2. runInference: pool round-robin can dispatch to a slot on a different GPU than the one the caller prepared inputs on. Detect foreign input GpuMats via cudaPointerGetAttributes and migrate them through a pinned host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D is unsupported and GeForce has no P2P). 3. warmUp: now pins its own CUDA device before uploading dummy inputs and returns success/failure; loadNetwork rejects engines whose warmup fails so broken slots can never join the pool silently. 4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an .aux0 engine-cache suffix so pre-workaround caches rebuild. Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for single-GPU bisection, and rich diagnostics on input-copy failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -207,6 +207,21 @@ std::string Engine<T>::serializeEngineOptions(const ANSCENTER::Options& options,
|
||||
+ std::to_string(CUDNN_MAJOR) + "."
|
||||
+ std::to_string(CUDNN_MINOR);
|
||||
|
||||
// -- Blackwell aux-stream workaround marker --------------------------------
|
||||
// On SM 12.x the builder forces maxAuxStreams=0 (see EngineBuildLoadNetwork.inl:
|
||||
// TRT 10.15 Myelin faults with concurrent contexts + aux streams). Tag those
|
||||
// engines so caches built *before* the workaround (with aux streams baked in)
|
||||
// miss and rebuild automatically instead of loading and crashing.
|
||||
#if NV_TENSORRT_MAJOR >= 10
|
||||
{
|
||||
cudaDeviceProp prop{};
|
||||
if (cudaGetDeviceProperties(&prop, options.deviceIndex) == cudaSuccess
|
||||
&& prop.major >= 12) {
|
||||
engineName += ".aux0";
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
return engineName;
|
||||
}
|
||||
|
||||
@@ -231,6 +246,32 @@ cv::cuda::GpuMat Engine<T>::blobFromGpuMats(const std::vector<cv::cuda::GpuMat>
|
||||
thread_local cv::cuda::GpuMat tl_blob;
|
||||
thread_local cv::cuda::GpuMat tl_floatImg;
|
||||
thread_local int tl_blobMaxElems = 0;
|
||||
thread_local int tl_blobDevice = -1;
|
||||
|
||||
// DEVICE AFFINITY: the cached GpuMats live on ONE device. When the same
|
||||
// thread serves engines on different GPUs (multi-GPU pool creation warms
|
||||
// probe on GPU A, then slots on GPU B), reusing a device-A blob with a
|
||||
// device-B stream causes invalid-argument launches and an async illegal
|
||||
// memory access that permanently poisons both CUDA contexts (sticky 700).
|
||||
// Detect the device switch and rebuild the cache on the current device.
|
||||
{
|
||||
int curDev = -1;
|
||||
cudaGetDevice(&curDev);
|
||||
if (curDev != tl_blobDevice) {
|
||||
if (tl_blobDevice >= 0) {
|
||||
// Free on the owning device — the async allocator frees via the
|
||||
// CURRENT device's null-stream pool, so switch, release, switch back.
|
||||
cudaSetDevice(tl_blobDevice);
|
||||
tl_blob.release();
|
||||
tl_floatImg.release();
|
||||
cudaSetDevice(curDev);
|
||||
ANS_DBG("TRT_Preproc", "blobFromGpuMats: device switch GPU[%d]->GPU[%d], preproc cache rebuilt",
|
||||
tl_blobDevice, curDev);
|
||||
}
|
||||
tl_blobMaxElems = 0;
|
||||
tl_blobDevice = curDev;
|
||||
}
|
||||
}
|
||||
|
||||
if (totalElems > tl_blobMaxElems) {
|
||||
tl_blob = cv::cuda::GpuMat(1, totalElems, CV_32FC1);
|
||||
|
||||
Reference in New Issue
Block a user