Commit Graph
109 Commits
Author SHA1 Message Date
nghianguyenandClaude Opus 5 86b520d85c ANSAWSS3: recover from dead sockets, and report why an upload failed
Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 21:24:12 +10:00
nghianguyenandClaude Fable 5 a4ea532f36 Fix multi-GPU inference on Blackwell (dual RTX 5080): four stacked fixes
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin,
zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has
a Myelin l2cm crash when creating a context on a second Blackwell device).

1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were
   device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1
   memory with a device-0 stream, causing an async illegal memory access
   that poisoned every CUDA context (sticky 700). Track owning device and
   rebuild the cache on device switch.

2. runInference: pool round-robin can dispatch to a slot on a different
   GPU than the one the caller prepared inputs on. Detect foreign input
   GpuMats via cudaPointerGetAttributes and migrate them through a pinned
   host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D
   is unsupported and GeForce has no P2P).

3. warmUp: now pins its own CUDA device before uploading dummy inputs and
   returns success/failure; loadNetwork rejects engines whose warmup fails
   so broken slots can never join the pool silently.

4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an
   .aux0 engine-cache suffix so pre-workaround caches rebuild.

Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for
single-GPU bisection, and rich diagnostics on input-copy failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-06 20:00:19 +10:00
nghianguyen b3436a61ee Prevent OOM 2026-06-18 05:55:31 +10:00
nghianguyen ebba70b9d9 Support Get log events 2026-05-24 09:19:24 +10:00
nghianguyen 47b80102a8 Add port to scanning devices 2026-05-24 08:33:34 +10:00
nghianguyen a2fa2c8ceb Support scanning ANSVIS servers device 2026-05-23 18:03:22 +10:00
nghianguyen 214f598e18 Support Generate Presigned URL for S3 2026-05-19 10:37:46 +10:00
nghianguyen 26f9b07ac1 Add adaptive padding to improve tracking 2026-05-14 22:01:07 +10:00
nghianguyen a43a99834a Complete RTMPOSE 2026-05-14 21:16:35 +10:00
nghianguyen 71404affb7 Correct logic in RTMPOSE when using generic model to detect person 2026-05-14 19:33:31 +10:00
nghianguyen 52ec13a8e3 improve pose 2026-05-14 19:17:09 +10:00
nghianguyen 8e4d9a9e5d Add person model into RTMPOSE 2026-05-14 19:05:20 +10:00
nghianguyen ee8f0d1163 Suppot RTMPOSE classes 2026-05-14 17:48:09 +10:00
nghianguyen a94bf88a5b Fix other ANSCV classes 2026-05-09 16:47:46 +10:00
nghianguyen aa323a07a3 Fix debug 2026-05-07 23:35:18 +10:00
nghianguyen 75693cadd9 Add jitter buffer 2026-05-07 22:36:27 +10:00
nghianguyen 2e0fee5e3d Fix VideoPlayer from stop after just few frames 2026-05-07 18:17:33 +10:00
nghianguyen a526aa6cdb Fix ALPR order issue 2026-05-06 19:55:52 +10:00
nghianguyen 66c67dff26 Fix onnx runtime loader for custom model 2026-05-05 16:23:01 +10:00
nghianguyen d03c2c1138 disable ALPR checker by default 2026-05-05 04:37:23 +10:00
nghianguyen 91d81cbd88 fix onnx yolo race condition 2026-05-04 23:45:05 +10:00
nghianguyen 03a63142ec Disable stabilized feature by default. Need ,more time to test 2026-05-04 09:39:07 +10:00
nghianguyen cff2a90fad Support multiple GPUs 2026-05-02 12:24:53 +10:00
nghianguyen fc9cf9671b Fix ANSONNXYOLO:
1. core/ANSLicensingSystem/ANSLicense.h:268 — added default initializers to ModelConfig POD members (inpWidth = 0, inpHeight = 0, etc.). This was the actual bug: CreateANSODHandle declares a local ModelConfig and only sets a few fields, leaving inpWidth/inpHeight as indeterminate stack memory. On your machine that memory consistently held ~80, slipping past the <=0 → 640 guard and driving the ORT session at 80×80 — which crashes YOLOv11's /model.12/Concat because 2·ceil(80/32)=6 ≠ ceil(80/16)=5.

2. modules/ANSODEngine/ANSONNXYOLO.h:75 — ONNXYOLO::setInputShape() now snaps width/height up to the next multiple of 32 (with a 64-pixel floor). Defense in depth: even if a future caller hands in a non-stride-32 size, the FPN concat alignment is preserved.

3. modules/ANSODEngine/ANSONNXYOLO.cpp — two diagnostic additions:

WarmUpEngine bails with a clear log line if the warm-up dims aren't a multiple of 32, instead of letting the model.12 error fire and mask the real cause.
transform() logs the input shape, canvas shape, source size, and dynamic flag once per session in DebugView — so any future shape-mismatch crash points at the offending HxW directly.
2026-05-01 20:24:08 +10:00
nghianguyen 1c8abf2a88 Fix AWS MinIO 2026-05-01 11:16:47 +10:00
nghianguyen 91efbb9a8f Fix AMD ALPR_OCR and custom model 2026-05-01 07:53:18 +10:00
nghianguyen e1f37ec767 Add ANSALPR country format 2026-04-28 19:23:46 +10:00
nghianguyen cd5d6f1923 Remove spaces, ".", and "-" from ALPR 2026-04-28 17:49:37 +10:00
nghianguyen 234f2c68a2 bucket prewarm + batch padding 2026-04-28 17:12:27 +10:00
nghianguyen dcf974c35c Fix AMD by changing from GetTensorData<T>() to GetTensorMutableData<T>() 2026-04-28 13:25:02 +10:00
nghianguyen da94e7f6da strip sensitive information from logs 2026-04-26 07:35:30 +10:00
nghianguyen bab94bf9bc Add log and change logger policy 2026-04-25 20:29:58 +10:00
nghianguyen ef2a122fec Add logs to custom models and model optimisation 2026-04-25 13:17:36 +10:00
nghianguyen 8a95ed6b8c add debug 2026-04-25 09:44:55 +10:00
nghianguyen 3a527d2dad Remove locks 2026-04-24 17:10:29 +10:00
nghianguyen fd2394a85a Fix deadlock 2026-04-24 12:54:16 +10:00
nghianguyen e2bf17289d Fix model extract race issue to all classes 2026-04-24 12:19:54 +10:00
nghianguyen baa88bcc48 Fix Concurrent extract + open race on train_last.onnx 2026-04-24 11:29:28 +10:00
nghianguyen 91bdb3f96b Fix play once video player 2026-04-23 08:36:52 +10:00
nghianguyen c625898f61 Fix Video Player that support generate black image when finish playing video (reach the end of video) 2026-04-23 08:08:34 +10:00
nghianguyen 6135649835 Use yuv and fix FR to use batch 4 2026-04-22 17:11:17 +10:00
nghianguyen a0d5c81814 Update 2026-04-22 16:52:40 +10:00
nghianguyen fc47a42c6a Use yuv and fix FR to use batch 4 2026-04-22 16:01:44 +10:00
nghianguyen 24d6d2e0a5 Use tracker and voting system to correct ALPR 2026-04-22 11:19:42 +10:00
nghianguyen 0e203eef99 Fix double stops 2026-04-22 10:19:03 +10:00
nghianguyen 57cc8e0a56 Fix double stop in ANSVideoPlayer 2026-04-22 10:10:16 +10:00
nghianguyen 97d814936d Enable log information. Disable NPU in U9 2026-04-21 15:48:27 +10:00
nghianguyen 7e772f76bc Improve ANSCV 2026-04-21 09:26:02 +10:00
nghianguyen 9f0a10a4c8 Improve ANSCV with sotfware decoder:
Thread-local staging Mat (video_player.cpp:1400-1407) — single biggest win. Eliminates the 12 MB per-call malloc/free cycle.
Contiguous get_buffer2 allocator (video_decoder.cpp:35-102) — keeps the 3 bulk memcpys cache-friendly. Would also enable FAST/zero-copy for resolutions where visible_h % 64 == 0.
SW-decoder thread config (video_decoder.cpp:528-540) — thread_count=0, thread_type=FRAME|SLICE. FRAME is downgraded to SLICE-only by AV_CODEC_FLAG_LOW_DELAY, but decode throughput is sufficient for your input rate.
SetTargetFPS(100) delivery throttle (already there) — caps onVideoFrame post-decode work at 10 FPS. Keeps the caller path warm-cached.
Instrumentation — [MEDIA_DecInit] / [MEDIA_Convert] / [MEDIA_SWDec] / [MEDIA_Timing] / [MEDIA_JpegTiming] — always-on regression detector, zero cost when ANSCORE_DEBUGVIEW=OFF.
2026-04-20 12:18:43 +10:00
nghianguyenandClaude Opus 4.7 adf32da2a2 Add early stale-out in ANSRTSPClient::GetImage
When the decoder hasn't produced a frame in 5s, skip the call to
_playerClient->getImage() entirely and return the cached frame with
unchanged _pts. LabVIEW sees STALE PTS one poll earlier and can
trigger reconnect sooner.

Threshold matches the existing checks on the duplicate-PTS branch and
in areImagesIdentical() so all three stale paths agree. Near-zero cost:
one getLastFrameAgeMs() call before the main path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 08:35:23 +10:00