A ServerOptimised zip is a per-GPU cache of a previous engine build. When
the engine profile changes (batch gate, TRT/CUDA upgrade), the cached
engine inside it goes stale: every re-extract (e.g. after Temp cleanup)
restores the old engine and costs a full ONNX rebuild, forever, because
the server only regenerates the zip when the file is missing.
Engine now records rebuiltFromOnnx() + engineFilePath() in
buildLoadNetwork (set only on the generate-from-ONNX branch; cache hits
and alternative-precision loads don't raise it). After a rebuild,
ANSFaceRecognizer::Initialize deletes the stale same-GPU engine variants
from the model folder and the supplied optimised zip, so the server's
next optimize pass recreates it with the fresh engine.
Safety: only zips whose parent folder is ServerOptimised/-ized qualify —
base model zips can never match. Engine purge is scoped to the
"<model>.engine.<GPU>." prefix, leaving other GPUs' engines intact on
multi-GPU machines. All deletes are best-effort (error_code); multi-GPU
pool paths never set the flag, so the purge is a no-op there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin,
zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has
a Myelin l2cm crash when creating a context on a second Blackwell device).
1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were
device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1
memory with a device-0 stream, causing an async illegal memory access
that poisoned every CUDA context (sticky 700). Track owning device and
rebuild the cache on device switch.
2. runInference: pool round-robin can dispatch to a slot on a different
GPU than the one the caller prepared inputs on. Detect foreign input
GpuMats via cudaPointerGetAttributes and migrate them through a pinned
host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D
is unsupported and GeForce has no P2P).
3. warmUp: now pins its own CUDA device before uploading dummy inputs and
returns success/failure; loadNetwork rejects engines whose warmup fails
so broken slots can never join the pool silently.
4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an
.aux0 engine-cache suffix so pre-workaround caches rebuild.
Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for
single-GPU bisection, and rich diagnostics on input-copy failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>