Fix multi-GPU inference on Blackwell (dual RTX 5080): four stacked fixes
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin, zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has a Myelin l2cm crash when creating a context on a second Blackwell device). 1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1 memory with a device-0 stream, causing an async illegal memory access that poisoned every CUDA context (sticky 700). Track owning device and rebuild the cache on device switch. 2. runInference: pool round-robin can dispatch to a slot on a different GPU than the one the caller prepared inputs on. Detect foreign input GpuMats via cudaPointerGetAttributes and migrate them through a pinned host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D is unsupported and GeForce has no P2P). 3. warmUp: now pins its own CUDA device before uploading dummy inputs and returns success/failure; loadNetwork rejects engines whose warmup fails so broken slots can never join the pool silently. 4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an .aux0 engine-cache suffix so pre-workaround caches rebuild. Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for single-GPU bisection, and rich diagnostics on input-copy failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1435,7 +1435,15 @@ trt_cache_create_context:
|
||||
// ============================================================================
|
||||
|
||||
if (m_verbose) std::cout << "\nInfo: Starting warm-up inference..." << std::endl;
|
||||
warmUp(m_verbose ? 10 : 1);
|
||||
if (!warmUp(m_verbose ? 10 : 1)) {
|
||||
// A failed warmup means this engine cannot run inference (wrong-device
|
||||
// buffers, poisoned context, ...). Report failure so pool creation
|
||||
// rejects the slot instead of silently adding a broken engine that
|
||||
// corrupts every subsequent inference dispatched to it.
|
||||
std::cout << "Error: Warm-up inference FAILED on GPU[" << m_options.deviceIndex
|
||||
<< "] -- engine rejected (loadNetwork returns false)" << std::endl;
|
||||
return false;
|
||||
}
|
||||
if (m_verbose) std::cout << "Info: Warm-up complete" << std::endl;
|
||||
|
||||
} // end of trt_cache_create_context scope
|
||||
@@ -1808,6 +1816,25 @@ bool Engine<T>::build(std::string onnxModelPath, const std::array<float, 3>& sub
|
||||
// Set hardware compatibility
|
||||
config->setHardwareCompatibilityLevel(nvinfer1::HardwareCompatibilityLevel::kNONE);
|
||||
|
||||
// -- Blackwell (SM 12.x) aux-stream workaround ----------------------------
|
||||
// On sm_120 (RTX 50-series), TRT 10.15's Myelin runtime faults when a second
|
||||
// concurrent execution context runs with auto-selected aux streams:
|
||||
// "Internal Error: MyelinCheckException ... l2cm_layer_internal.cpp:174"
|
||||
// followed by CUDA error 700 (illegal memory access) poisoning the context.
|
||||
// Aux streams only provide intra-inference parallelism (irrelevant for the
|
||||
// small per-slot models we run); throughput comes from the engine pool's
|
||||
// one-context-per-slot design, which is unaffected. Force 0 aux streams on
|
||||
// Blackwell until NVIDIA fixes the Myelin multi-stream path.
|
||||
// NOTE: baked in at build time -- stale cached .engine files must be deleted
|
||||
// for this to take effect.
|
||||
#if NV_TENSORRT_MAJOR >= 10
|
||||
if (prop.major >= 12) {
|
||||
config->setMaxAuxStreams(0);
|
||||
std::cout << "Aux streams disabled on SM " << prop.major << "." << prop.minor
|
||||
<< " (Blackwell Myelin multi-stream workaround)" << std::endl;
|
||||
}
|
||||
#endif
|
||||
|
||||
// -- TensorRT 10+ tactic DRAM pool ----------------------------------------
|
||||
// Separate scratch pool for kernel-selection during build. Without this,
|
||||
// tactic evaluation competes with the workspace allocation and tactics
|
||||
|
||||
Reference in New Issue
Block a user