Fix multi-GPU inference on Blackwell (dual RTX 5080): four stacked fixes
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin, zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has a Myelin l2cm crash when creating a context on a second Blackwell device). 1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1 memory with a device-0 stream, causing an async illegal memory access that poisoned every CUDA context (sticky 700). Track owning device and rebuild the cache on device switch. 2. runInference: pool round-robin can dispatch to a slot on a different GPU than the one the caller prepared inputs on. Detect foreign input GpuMats via cudaPointerGetAttributes and migrate them through a pinned host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D is unsupported and GeForce has no P2P). 3. warmUp: now pins its own CUDA device before uploading dummy inputs and returns success/failure; loadNetwork rejects engines whose warmup fails so broken slots can never join the pool silently. 4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an .aux0 engine-cache suffix so pre-workaround caches rebuild. Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for single-GPU bisection, and rich diagnostics on input-copy failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -731,7 +731,7 @@ int ANSVISTest() {
|
||||
|
||||
//std::string videoFilePath = "E:\\Programs\\DemoAssets\\Videos\\ANSVIS_Issues\\face.mp4";
|
||||
//std::string videoFilePath = "E:\\Programs\\DemoAssets\\Videos\\TestFR\\Face31.mp4";
|
||||
std::string videoFilePath = "E:\\Programs\\DemoAssets\\Videos\\classroom.mp4";
|
||||
std::string videoFilePath = "C:\\Programs\\DemoAssets\\Videos\\classroom.mp4";
|
||||
//std::string videoFilePath = "E:\\Programs\\DemoAssets\\Videos\\TestFR\\school1.mp4";// "C:\\ProgramData\\ANSCENTER\\Shared\\classroom.mp4";
|
||||
|
||||
const char* configFilePath = "";
|
||||
@@ -1713,15 +1713,31 @@ int SCRFDVideoRepro()
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main()
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
// ── GPU visibility bisect (Blackwell Myelin l2cm debugging) ─────────────
|
||||
// Usage: ANSFR-UnitTest.exe --gpu 0 → only GPU0 visible (Test A, display GPU)
|
||||
// ANSFR-UnitTest.exe --gpu 1 → only GPU1 visible (Test B, headless GPU)
|
||||
// ANSFR-UnitTest.exe → all GPUs (default behaviour)
|
||||
// Sets CUDA_VISIBLE_DEVICES before the first CUDA call in this process, so
|
||||
// the CUDA runtime only enumerates the chosen physical GPU (it becomes
|
||||
// device 0 inside the process).
|
||||
for (int i = 1; i + 1 < argc; ++i) {
|
||||
if (std::string(argv[i]) == "--gpu") {
|
||||
_putenv_s("CUDA_VISIBLE_DEVICES", argv[i + 1]);
|
||||
std::cout << "==============================================\n"
|
||||
<< " GPU BISECT: CUDA_VISIBLE_DEVICES=" << argv[i + 1] << "\n"
|
||||
<< " (only this physical GPU is visible to CUDA)\n"
|
||||
<< "==============================================" << std::endl;
|
||||
}
|
||||
}
|
||||
|
||||
//FaceDetectorTest();
|
||||
//TestFaceRecognition();
|
||||
//TestCompleteFR1();
|
||||
|
||||
//ANSVISImageTest();
|
||||
//ANSVISTest();
|
||||
ANSVISTest();
|
||||
//ANSVISTestFilePlayer();
|
||||
// ANSVISRecordingTest();
|
||||
//FRStressTest();
|
||||
@@ -1735,7 +1751,7 @@ int main()
|
||||
// TestCompleteFR();
|
||||
//TestFaceRecognition();
|
||||
//FaceRecognitionBenchmark();
|
||||
SCRFDVideoRepro();
|
||||
//SCRFDVideoRepro();
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user