Commit Graph
18 Commits
Author SHA1 Message Date
nghianguyenandClaude Fable 5 a4ea532f36 Fix multi-GPU inference on Blackwell (dual RTX 5080): four stacked fixes
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin,
zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has
a Myelin l2cm crash when creating a context on a second Blackwell device).

1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were
   device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1
   memory with a device-0 stream, causing an async illegal memory access
   that poisoned every CUDA context (sticky 700). Track owning device and
   rebuild the cache on device switch.

2. runInference: pool round-robin can dispatch to a slot on a different
   GPU than the one the caller prepared inputs on. Detect foreign input
   GpuMats via cudaPointerGetAttributes and migrate them through a pinned
   host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D
   is unsupported and GeForce has no P2P).

3. warmUp: now pins its own CUDA device before uploading dummy inputs and
   returns success/failure; loadNetwork rejects engines whose warmup fails
   so broken slots can never join the pool silently.

4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an
   .aux0 engine-cache suffix so pre-workaround caches rebuild.

Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for
single-GPU bisection, and rich diagnostics on input-copy failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:00:19 +10:00
nghianguyen b3436a61ee Prevent OOM 2026-06-18 05:55:31 +10:00
nghianguyen 91efbb9a8f Fix AMD ALPR_OCR and custom model 2026-05-01 07:53:18 +10:00
nghianguyen dcf974c35c Fix AMD by changing from GetTensorData<T>() to GetTensorMutableData<T>() 2026-04-28 13:25:02 +10:00
nghianguyen ef2a122fec Add logs to custom models and model optimisation 2026-04-25 13:17:36 +10:00
nghianguyen 57cc8e0a56 Fix double stop in ANSVideoPlayer 2026-04-22 10:10:16 +10:00
nghianguyen 97d814936d Enable log information. Disable NPU in U9 2026-04-21 15:48:27 +10:00
nghianguyen 7e772f76bc Improve ANSCV 2026-04-21 09:26:02 +10:00
nghianguyenandClaude Opus 4.6 8d2db78cee Remove [Engine] and [EnginePoolManager] debug log messages
Cleaned up verbose engine telemetry emitted to stdout/stderr and the
Windows Event Viewer. Removes logEngineEvent/logEvent calls (and their
diagnostic-only locals) across the TensorRT engine load, build, run,
multi-GPU, and pool-manager paths, plus the now-unused logEvent helper
in EnginePoolManager.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 18:09:08 +10:00
nghianguyen 5706615ed5 Support tracker to improve ALPR_OCR 2026-04-14 21:18:10 +10:00
nghianguyen f9a0af8949 Improve ALPR_OCR peformance 2026-04-14 20:30:21 +10:00
nghianguyen 257145c429 Fix Intel ONNX runtime 2026-04-10 17:13:47 +10:00
nghianguyen 69787b0ff0 Fix AMD and OpenVINO 2026-04-08 13:45:52 +10:00
nghianguyen 98681f4da6 Use CPU resize before upload to GPU to remove PCIe bottleneck 2026-04-04 22:29:08 +11:00
nghianguyen e134ebdf15 Use software decoder by default 2026-04-04 20:19:54 +11:00
nghianguyen 958cab6ae3 Fix NV12 crash issue when recreate camera object 2026-04-02 22:07:27 +11:00
nghianguyen 01eabf76bd Fix ANSOCR TensorRT Release 2026-03-30 09:24:04 +11:00
nghianguyen 7b4134133c Initial setup for CLion 2026-03-28 16:54:11 +11:00