Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.
Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
pooling it (23 sites). An HTTP error status is not a transport failure - the
server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
the configured one, 5 s by default); a fresh connection keeps the full one.
A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
once at startup, so returning left the handle dead for the process lifetime.
Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
completing just after SetAuthentication cleared the pool cannot leave an
unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
query params, and the gzip Content-Encoding set by UploadTextData. Not
ClearAllHeaders - that wipes SigV4 state (fcc985e).
Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.
Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.
Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The id is built from up to 6 enumerated MACs. On Linux getifaddrs() also
returns virtual interfaces — docker0 and the veth* pairs each container
creates, tun*/wg* from a VPN, br-*/virbr* bridges. Container veths carry
RANDOM MACs and appear/disappear with the containers, so the enumerated set
can differ between calls: the hardware id changes and an activated licence
silently stops matching its machine. Boxes running Docker (the Coder feature)
would hit this routinely.
This matters more on Linux than elsewhere because two of the four id
components are already constant there: the system-UUID slot is a literal and
GetHDDPropertyList is an empty function on non-Windows. The id rests on CPU id
+ MACs, so unstable MACs are most of the instability.
Filter mirrors what the Windows branch already does via WMI (PNPDeviceId LIKE
%PCI% — physical adapters only). The test is /sys/class/net/<name>/device,
the kernel's own record of a hardware-backed interface, so it stays correct as
new virtual-interface flavours appear; a name-prefix denylist is kept only for
environments without /sys. MACs are bucketed during the walk and chosen after
it, so a veth appearing earlier in the list cannot consume a slot and push a
real NIC out.
FAIL-SAFE: if no physical NIC is found, fall back to the unfiltered set. An
empty list would drop all MAC entropy and — with the UUID and disk slots
already constant — leave near-identical ids across different machines.
Windows and macOS/BSD paths are untouched (both #else branches preserve the
existing behaviour exactly). Logic verified against a simulated Docker host:
enp3s0/wlp2s0 accepted; lo, docker0, veth*, br-*, tun0 rejected.
ORDERING: this CHANGES the id on any Linux box that has virtual interfaces, so
it must land BEFORE the first Linux licences are issued — afterwards it would
invalidate them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GetCurrentHardwareId sized its bit stream at 125 bits (16 bytes) but
writes 147 (3 version + 9 x 16-bit hashes), so the last two hash slots
overflowed the heap block by 3 bytes on every id computation. The
Windows heap absorbed it silently for years; it SIGSEGVs under macOS
malloc — found when ANSGateway enabled this code on POSIX (fixed there
in ANSGateway 3a34c68/6c3cdfd; this is the upstream port).
Two parts, both id-preserving (only the first 16 bytes feed the base32
encode — verified byte-identical ids before/after in the gateway):
- hwid.cpp: size the stream to the 147 bits actually written.
- BitStream3: bounds-safe byteAt/setByte accessors on every Write/Read,
Attach records the real buffer size, default ctor zero-inits — an
under-sized stream can no longer corrupt the heap, and out-of-range
reads return 0 (identical to the base32 padding zeros the Match path
always read there).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Overnight soak revealed the gap: ANSRTSPClient does not auto-reconnect
by design (the application layer polls staleness and calls
ReconnectRTSP - ANSVIS/LabVIEW does this in production), but the unit
test never did, so when the laptop's Modern Standby killed the TCP
session at ~1 AM the client returned the cached frame until morning.
Poll GetRTSPLastFrameAgeMs each loop; at 20 s without a decoded frame,
call ReconnectRTSP (timestamped console line), retrying at most every
15 s. Threshold sits above the corruption keyframe gate's ~12 s
worst-case hold so a reconnect never preempts an in-flight recovery.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a sequence gap the H.264/H.265 depacketizers resumed reassembly at
the next NAL/FU start. On multi-slice streams every slice starts a NAL,
so reassembly resumed MID-FRAME and delivered access units missing their
leading slices — manufacturing "First slice in a frame missing" /
"Ignoring POC change between slices" / "Could not find ref" decoder
errors and green/blocky concealment output far beyond the actual loss.
Resume only at a true AU boundary instead:
H.265 - non-VCL NAL (type >= 32) or VCL slice with
first_slice_segment_in_pic_flag set (first bit after the
2-byte NAL header)
H.264 - non-VCL NAL (type >= 6) or VCL slice with
first_mb_in_slice == 0 (ue(v): MSB set after 1-byte header)
Also: stderr visibility for RTP sequence gaps (rtp_rx logs went only to
the sys_log file) and for av_log-bridge latches in video_decoder — the
global av_log level filters decoder errors from the console, which made
keyframe-gate arms look causeless during diagnosis.
Live verification, HEVC camera on lossy Wi-Fi (NVDEC decode):
before: 271 decoder errors / 3 min, delivery collapsed to ~2 fps
after: 3 errors / 2 min, full ~21 fps, zero corrupt frames displayed
Remaining rare errors show POC ref misses with ZERO RTP sequence gaps,
i.e. pre-decode queue eviction under TCP burst clumps - a separate,
much smaller issue absorbed invisibly by the keyframe gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
log_print writes to the sys_log file (unopened in unit tests / field
debugging), so the gate looked inert next to FFmpeg's stderr error lines.
Emit arm/release one-liners to stderr alongside them.
Live verification during a severe Wi-Fi loss burst (271 HEVC errors /
3 min, NVDEC path): gate armed 19x, released 17x, dropping 29-118
corrupt frames per burst and resuming on the next clean keyframe.
The av_log bridge (273dd44) is confirmed working end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The corruption re-arm added in 6344062 keyed off AVFrame::decode_error_flags,
but the HEVC decoder on the hwaccel (NVDEC) path never sets it: 120 s of live
Wi-Fi packet loss produced 10+ "First slice in a frame missing" / "Could not
find ref with POC n" log bursts with decode_error_flags == 0 on every frame,
so the keyframe gate never fired.
Bridge: install a process-wide av_log callback (chains to
av_log_default_callback) that maps the logging AVCodecContext back to its
CVideoDecoder and latches a per-decoder m_bCorruptSeen flag on AV_LOG_ERROR
messages. readFrame() consumes the latch alongside decode_error_flags to
re-arm the keyframe gate. With hwaccel the decoder is single-threaded, so
the logging context is exactly our AVCodecContext; SW frame-threaded paths
log from per-thread clones but set decode_error_flags instead, which the
existing check covers.
Registered after avcodec_open2, unregistered in uninit() before the context
is freed. Map lookup only on error-level messages (rare). Also promote the
gate-arm message to log_print(HT_LOG_WARN) for console visibility.
Clean-path verified live: 6105 frames / 5 min NVDEC RTSP with zero errors
and no regressions. Loss-path verification pending a naturally lossy window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restores the previously shelved decoder/player hardening and extends it:
1. Startup keyframe gating (video_decoder): after (re)start the decoder
joins the stream mid-GOP; frames referencing unseen pictures render as
green/blocky concealment garbage. Drop everything until the first IDR
(300-frame safety valve so unmarked streams can't blank forever).
2. NEW - corruption re-arm: on lossy links (Wi-Fi cameras) dropped slices
surface as "First slice in a frame missing" / "Could not find ref with
POC n" and FFmpeg marks the output via AVFrame::decode_error_flags.
Re-arm the keyframe gate and drop the rest of the GOP so the display
holds the last clean frame instead of showing concealment artifacts;
corrupt drops count toward the safety valve.
3. libyuv NV12->BGR conversion (video_player): honors NVDEC/D3D11VA padded
strides (1080->1088 rows) that cv::cvtColorTwoPlane mishandled (green
cast + striped noise), picks BT.601/709/2020 coefficients from the
frame's signalled colorspace, and is ~3-5x faster than sws_scale.
Verified live against an RTSP HEVC camera (NVDEC path): steady frame flow,
no regressions on the clean-stream path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The empty-frame path slept 1 s without calling cv::waitKey, so the
HighGUI window stopped pumping Windows messages and went "Not
Responding" whenever the RTSP stream stalled for a few seconds.
Wait via cv::waitKey(1000) instead (ESC still exits). Same fix in
all three test loops that shared the pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two-pass slot acquisition in runInferenceFromPool: Pass A picks an idle
slot whose deviceIndex matches the input GpuMats' resident device
(zero-copy dispatch); Pass B falls back to plain round-robin, where
runInference migrates the inputs via the pinned-host two-hop copy.
Callers that prepare frames on GPU A no longer pay the ~1 ms/frame PCIe
migration when a same-GPU slot is idle, while spillover to the other GPU
under load still works. Input residency is queried outside the slot
mutex via cudaPointerGetAttributes (cheap driver lookup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified on dual RTX 5080: 355/355 detections, both GPUs in round-robin,
zero CUDA errors. Requires TensorRT >= 10.16.1 in SharedCore (10.15.1 has
a Myelin l2cm crash when creating a context on a second Blackwell device).
1. blobFromGpuMats: thread_local GpuMat caches (tl_blob/tl_floatImg) were
device-blind — a pool thread warming GPU[1] then GPU[0] reused device-1
memory with a device-0 stream, causing an async illegal memory access
that poisoned every CUDA context (sticky 700). Track owning device and
rebuild the cache on device switch.
2. runInference: pool round-robin can dispatch to a slot on a different
GPU than the one the caller prepared inputs on. Detect foreign input
GpuMats via cudaPointerGetAttributes and migrate them through a pinned
host staging buffer (two-hop D2H+H2D; direct cross-device cudaMemcpy2D
is unsupported and GeForce has no P2P).
3. warmUp: now pins its own CUDA device before uploading dummy inputs and
returns success/failure; loadNetwork rejects engines whose warmup fails
so broken slots can never join the pool silently.
4. Blackwell builder workaround: setMaxAuxStreams(0) on SM 12.x plus an
.aux0 engine-cache suffix so pre-workaround caches rebuild.
Also: ANSFR-UnitTest gains --gpu N flag (sets CUDA_VISIBLE_DEVICES) for
single-GPU bisection, and rich diagnostics on input-copy failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add ConsoleTest.cpp, a headless (no-GUI) smoke test for ANSOPENCV image
ops plus an optional RTSP capture path, built as its own executable
(ANSCV-ConsoleTest) so it does not collide with ANSCV-UnitTest's GUI main.
Make the ans_imshow.hpp include (shipped with ANSMIND, outside the repo)
configurable via the ANSMIND_INCLUDE_DIR cache variable, defaulting to
C:/ProgramData/ANSCENTER/ANSMIND. Warn rather than fail configure when the
header is absent so unrelated targets still build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1. core/ANSLicensingSystem/ANSLicense.h:268 — added default initializers to ModelConfig POD members (inpWidth = 0, inpHeight = 0, etc.). This was the actual bug: CreateANSODHandle declares a local ModelConfig and only sets a few fields, leaving inpWidth/inpHeight as indeterminate stack memory. On your machine that memory consistently held ~80, slipping past the <=0 → 640 guard and driving the ORT session at 80×80 — which crashes YOLOv11's /model.12/Concat because 2·ceil(80/32)=6 ≠ ceil(80/16)=5.
2. modules/ANSODEngine/ANSONNXYOLO.h:75 — ONNXYOLO::setInputShape() now snaps width/height up to the next multiple of 32 (with a 64-pixel floor). Defense in depth: even if a future caller hands in a non-stride-32 size, the FPN concat alignment is preserved.
3. modules/ANSODEngine/ANSONNXYOLO.cpp — two diagnostic additions:
WarmUpEngine bails with a clear log line if the warm-up dims aren't a multiple of 32, instead of letting the model.12 error fire and mask the real cause.
transform() logs the input shape, canvas shape, source size, and dynamic flag once per session in DebugView — so any future shape-mismatch crash points at the offending HxW directly.
Thread-local staging Mat (video_player.cpp:1400-1407) — single biggest win. Eliminates the 12 MB per-call malloc/free cycle.
Contiguous get_buffer2 allocator (video_decoder.cpp:35-102) — keeps the 3 bulk memcpys cache-friendly. Would also enable FAST/zero-copy for resolutions where visible_h % 64 == 0.
SW-decoder thread config (video_decoder.cpp:528-540) — thread_count=0, thread_type=FRAME|SLICE. FRAME is downgraded to SLICE-only by AV_CODEC_FLAG_LOW_DELAY, but decode throughput is sufficient for your input rate.
SetTargetFPS(100) delivery throttle (already there) — caps onVideoFrame post-decode work at 10 FPS. Keeps the caller path warm-cached.
Instrumentation — [MEDIA_DecInit] / [MEDIA_Convert] / [MEDIA_SWDec] / [MEDIA_Timing] / [MEDIA_JpegTiming] — always-on regression detector, zero cost when ANSCORE_DEBUGVIEW=OFF.
When the decoder hasn't produced a frame in 5s, skip the call to
_playerClient->getImage() entirely and return the cached frame with
unchanged _pts. LabVIEW sees STALE PTS one poll earlier and can
trigger reconnect sooner.
Threshold matches the existing checks on the duplicate-PTS branch and
in areImagesIdentical() so all three stale paths agree. Near-zero cost:
one getLastFrameAgeMs() call before the main path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
getImage() previously held _mutex across the 4K NV12->BGR sws_scale in
avframeToCVMat, blocking the decoder callback (onVideoFrame) for 100-300ms
per frame. Under multi-camera load this cascaded into 5-21s frame stalls
and STALE PTS events in the log.
- avframeToCVMat: drop outer _mutex. NV12/YUV420P paths touch no shared
state; avframeAnyToCvmat still locks internally for swsCtx.
- getImage: split into two short locked phases with the BGR conversion
unlocked between them. Decoder callbacks can push new frames and run
the CUDA HW capture path in parallel with the reader's conversion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removing the Fix 1 ClearAllHeaders/ClearAllQueryParams block from all 8
upload entry points. CkRest.AddHeader already overwrites by header name,
so the clears were unnecessary; they also wipe the internal state
Chilkat's SigV4 signer relies on between requests, which caused AWS to
reject every PUT with HTTP 400 MaxMessageLengthExceeded (max 35240 B).
Per-step ClearAllQueryParams inside multipart kept (followed immediately
by AddQueryParam).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleaned up verbose engine telemetry emitted to stdout/stderr and the
Windows Event Viewer. Removes logEngineEvent/logEvent calls (and their
diagnostic-only locals) across the TensorRT engine load, build, run,
multi-GPU, and pool-manager paths, plus the now-unused logEvent helper
in EnginePoolManager.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- NvJpegPool: singleton pool of 4 NvJpegCompressor instances with
lock-free slot acquisition (~160MB VRAM). Threads that can't grab
a slot fall back to TurboJPEG with zero wait.
- JPEG passthrough: BmpToJpeg now checks if input is already JPEG
(FF D8 FF magic) and copies directly without re-encoding.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BmpToJpeg was slow (~25-45ms for 4K) due to two bottlenecks:
1. cv::imdecode for BMP parsing (unnecessary for uncompressed BMP)
2. TurboJPEG CPU encoding (~11ms for 4K)
Fix 1: Zero-copy BMP parsing — parse header directly and wrap pixel
data in cv::Mat without allocation or copy. Eliminates ~47MB of heap
allocations per 4K frame.
Fix 2: NvJpegCompressor class using nvJPEG hardware encoder on NVIDIA
GPUs (~1-2ms for 4K). Integrated into CompressJpegToString so all 5
JPEG encoding callsites benefit automatically. Reusable GPU buffer
avoids per-frame cudaMalloc/cudaFree. Silent fallback to TurboJPEG
on Intel/AMD or if nvJPEG fails.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix 1 — Chunk oversized bucket groups (the correctness fix)
ONNXOCRRecognizer::RecognizeBatch now slices each bucket group into chunks of ≤ kRecMaxBatch before submitting to TRT. A frame with 30 crops in bucket 320 produces two back-to-back batched calls (24 + 6), both within the profile, both on the fast path.
Fix 2 — Raise the profile max from 16 to 24 (the performance fix)
The old profile max was 16; your real scenes routinely hit 24. Raising the profile max to 24 means the common 12-plate scene (24 crops) fits in a single batched call with no chunking needed. Scenes with > 24 crops now use chunking, but that's rare.