Commit Graph
8 Commits
Author SHA1 Message Date
nghianguyenandClaude Opus 5 86b520d85c ANSAWSS3: recover from dead sockets, and report why an upload failed
Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 21:24:12 +10:00
nghianguyen bc1517598d Plans for CMS 2026-06-12 14:13:26 +10:00
nghianguyen 4678d87e33 Add plan to port to linux 2026-05-23 17:29:57 +10:00
nghianguyen dcf974c35c Fix AMD by changing from GetTensorData<T>() to GetTensorMutableData<T>() 2026-04-28 13:25:02 +10:00
nghianguyen f81737ffc6 Add debug and log functions to ANSLIB 2026-04-26 06:59:42 +10:00
nghianguyen 750ccff58b Plan ffmpeg migration v1 2026-04-15 23:51:30 +10:00
nghianguyen 7ea9fd0923 Plan ffmpeg migration 2026-04-15 23:38:57 +10:00
nghianguyen 091a61d2be Fix Window search path for ANSLIB 2026-04-06 08:31:26 +10:00