Fix AMD ALPR_OCR and custom model

This commit is contained in:
2026-05-01 07:53:18 +10:00
parent e1f37ec767
commit 91efbb9a8f
25 changed files with 2841 additions and 429 deletions
+67 -7
View File
@@ -210,14 +210,74 @@ namespace ANSCENTER {
int NUM_KPS{ 0 };
float KPS_THRESHOLD{ 0.5f };
// ONNX Runtime inference engine
std::unique_ptr<ONNXYOLO> m_ortEngine;
// ONNX Runtime inference engine.
//
// B2: shared_ptr (was unique_ptr) so concurrent inference threads
// can grab a ref-counted local copy under the brief snapshot lock
// and dereference it OUTSIDE the lock without racing with a
// concurrent reset() in the DML-fatal CPU-fallback path. Without
// this, a thread doing `m_ortEngine->detect(...)` lock-free could
// see its underlying object freed mid-call by another thread that
// caught a DML E_FAIL and rebuilt the session — that was the
// 0xc0000005 access violation observed in ANSODEngine.dll under
// AMD/DirectML when multiple AI tasks ran in parallel.
std::shared_ptr<ONNXYOLO> m_ortEngine;
// DML device-lost recovery: when DirectML's GPU device is removed
// (HRESULT 887A0005), the session is permanently broken. We detect
// this once, attempt a CPU-fallback recreation, and suppress further
// error-log flooding.
bool _dmlDeviceLost{ false };
// ────────────────────────────────────────────────────────────────
// DML stay-on-GPU recovery state.
//
// AMD's DirectML EP intermittently throws 80004005 (E_FAIL) from
// DmlGraphFusionHelper / DmlCommandRecorder when the driver
// chokes under concurrent inference pressure. Empirically this
// is NOT a permanent device loss — recreating the ORT session on
// the same DML EP clears the wedged state and resumes GPU
// inference at full speed. Only the truly-fatal 887A0005
// (DXGI_ERROR_DEVICE_REMOVED) requires CPU fallback.
//
// Recovery contract (used by every Run() catch block):
// 1. compare_exchange on _dmlRecreateInProgress elects one
// thread as the recreate owner. Other threads return {}
// and retry on the next call.
// 2. The owner checks the cooldown — if we recreated in the
// last kDmlRecreateCooldownMs ms, skip (driver is in a
// prolonged bad state, don't thrash).
// 3. If error is IsDmlDeviceRemoved (887A0005), or we've hit
// kMaxConsecutiveDmlFailures recreates without a successful
// inference in between, fall back to CPU as last resort.
// 4. Otherwise reset() + InitOrtEngine(<original DML EP>) +
// WarmUpEngine() to rebuild on the same GPU EP.
// 5. On successful inference, _dmlConsecutiveFailures resets
// to 0, so isolated transient hiccups don't accumulate
// toward the CPU-fallback threshold.
//
// This replaces the old single-shot `bool _dmlDeviceLost` flag
// which latched on the first failure and forced CPU forever —
// user requirement is to keep AI tasks running on GPU, not to
// trade GPU performance for stability the moment the driver
// hiccups once.
// ────────────────────────────────────────────────────────────────
std::atomic<bool> _dmlRecreateInProgress{ false };
std::atomic<int> _dmlConsecutiveFailures{ 0 };
std::chrono::steady_clock::time_point _lastDmlRecreateAt{};
// Tunables — tuned for AMD Radeon 780M observed behavior.
// threshold: catches isolated transient AMD DML hiccups. After
// 3 consecutive recreates without a successful inference, the
// driver is in a prolonged bad state and CPU is the only
// forward path.
// cooldown: prevents recreate thrash. AMD DML session creation
// takes 200–500 ms; if E_FAIL fires from many threads at once
// we don't want to recreate N times in a row.
static constexpr int kMaxConsecutiveDmlFailures = 3;
static constexpr int64_t kDmlRecreateCooldownMs = 5000;
// Centralised DML-fatal recovery used by all 3 catch sites.
// Caller must have already determined the exception is DML-related
// (IsDmlFatalError == true). msg is the exception message used
// both to log and to detect the device-removed sub-case. Returns
// immediately without doing anything when another thread already
// owns the recreate (CAS loser path).
void HandleDmlRunFailure(const char* context, const std::string& msg);
// Internal detection pipeline
std::vector<Object> DetectObjects(const cv::Mat& inputImage,