Commit Graph
9 Commits
Author SHA1 Message Date
nghianguyenandClaude Opus 5 86b520d85c ANSAWSS3: recover from dead sockets, and report why an upload failed
Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 21:24:12 +10:00
nghianguyen 214f598e18 Support Generate Presigned URL for S3 2026-05-19 10:37:46 +10:00
nghianguyen 1c8abf2a88 Fix AWS MinIO 2026-05-01 11:16:47 +10:00
nghianguyen fd2394a85a Fix deadlock 2026-04-24 12:54:16 +10:00
nghianguyen 3418090042 - Remove ALPR_OD Tracker and voting system.
- Disable debugview
2026-04-19 23:15:50 +10:00
nghianguyen a47ea25e8b Fix Image management (BMP, BMPtoJPEG) and remove debug info from AWS Upload 2026-04-17 20:59:21 +10:00
nghianguyenandClaude Opus 4.7 fcc985efb9 Fix AWS S3 upload failure: remove ClearAllHeaders that was wiping SigV4 state
Removing the Fix 1 ClearAllHeaders/ClearAllQueryParams block from all 8
upload entry points. CkRest.AddHeader already overwrites by header name,
so the clears were unnecessary; they also wipe the internal state
Chilkat's SigV4 signer relies on between requests, which caused AWS to
reject every PUT with HTTP 400 MaxMessageLengthExceeded (max 35240 B).
Per-step ClearAllQueryParams inside multipart kept (followed immediately
by AddQueryParam).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 08:13:32 +10:00
nghianguyen a63feab0ff Ensure the threadsafe for video upload 2026-04-17 07:03:03 +10:00
nghianguyen 8a2e721058 Refactor project structure 2026-03-28 19:56:39 +11:00