ANSAWSS3: recover from dead sockets, and report why an upload failed

Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 21:24:12 +10:00
co-authored by Claude Opus 5
parent 6f1d069d14
commit 86b520d85c
4 changed files with 606 additions and 206 deletions
+79
View File
@@ -0,0 +1,79 @@
# ANSAWSS3 — return codes and retry behaviour
Reference for callers of the `*ANSAWSHandle` exports in `ANSUtilities.dll`
(LabVIEW, C++, .NET). Source of truth: `modules/ANSUtilities/ANSUtilities.h`
(`kCall*` constants) and `modules/ANSUtilities/dllmain.cpp`.
## The one rule
**`rc == 1` means success. Everything else is a failure.** Success is the only
positive value, so `rc == 1` and `rc != 1` are both always correct tests. The
negative values exist so a caller can tell *why* it failed, and therefore
whether retrying is worthwhile.
## Upload functions
Applies to every `Upload*ANSAWSHandle`, `Upload*ANSAWSHandle_CPP` and
`Upload*ANSAWSHandle_V2` export (JPEG, binary, text, file-stream, multipart,
with or without prefix).
| Code | Meaning | Retry? |
|-----:|---------|--------|
| `1` | Uploaded. `uploadedFilePath` holds the object URL. | — |
| `0` | A required pointer argument was `NULL`. | **No** — caller bug |
| `-1` | Handle is not in the registry: never created, already released, or a stale value. | **No** — re-create the handle |
| `-2` | Failed, reason not classified. Should not occur; kept as a fallback. | Yes |
| `-3` | A `std::exception` escaped. | Yes |
| `-4` | An unknown exception escaped. | Yes |
| `-20` | **Permanent.** Bad argument (empty bucket/filename, part size < 5 MB, unreadable file), invalid licence or unlock code, or S3 answered with a 4xx — wrong bucket, no permission, bad key. | **No** — fix the configuration; retrying burns time and changes nothing |
| `-21` | **Not connected.** `Connect()` has not succeeded yet, or the link went down and the background retry thread is still working on it. | **Yes, later** — back off a few seconds |
| `-22` | **Transient.** Network error, timeout, or HTTP 5xx, still failing after the DLL's own internal retries. | **Yes** |
A practical LabVIEW policy:
* `rc = 1` → done.
* `rc = -20` or `0` → log it and stop; it will never succeed as-is.
* `rc = -21` → queue the image and try again in a few seconds.
* anything else negative → retry, with backoff.
Note that `-22` already means the DLL retried internally (up to 20 attempts
within a 30 s budget) before giving up, so an outer retry should not be tight.
## Connect
`ConnectANSAWSHandle` / `_V2`:
| Code | Meaning |
|-----:|---------|
| `1` | Connected; ready for uploads once `SetAuthentication` is called. |
| `0` | A required argument was `NULL`, the handle is unknown, the licence is invalid, or the endpoint resolves but refuses the connection. |
| `2` | The endpoint could not be resolved at all (no internet). A background thread keeps retrying — 3 s, doubling to a 30 s ceiling — until it connects or the handle is released. **You do not need to call `Connect` again.** Uploads return `-21` in the meantime. |
## Other functions
`CreateANSAWSHandle`, `ReleaseANSAWSHandle`, `SetAuthenticationANSAWSHandle`,
`SetProxyANSAWSHandle`, the `ListBucket*` / `CreateBucket*` / `DeleteBucket*`
family, `DownloadFileStreamANSAWSHandle` and `GeneratePresignedURLANSAWSHandle`
return `1` on success and `0` on failure. `GeneratePresignedURLANSAWSHandle`
(the `LStrHandle` variant) additionally uses `-1`, `-2`, `-3`, `-4` with the
same meanings as the table above.
## How connections behave
* `SetAuthentication` opens and keeps 8 pooled HTTPS connections; uploads borrow
and return them, so the TLS handshake is not repeated per image.
* A connection whose **transport** failed is closed, never returned to the pool.
A connection that merely received an HTTP error status is healthy and is kept.
* A pooled connection idle longer than 15 s is closed and replaced rather than
handed out, since middleboxes silently drop idle flows.
* The first request on a reused connection uses a short idle timeout (a quarter
of the configured one, 5 s by default); a fresh connection gets the full
timeout. This bounds what a dead socket costs.
* One upload call is capped at 30 s of wall clock across all internal retries.
## Diagnostics
`ANSAWSS3.log` and DebugView (`AWSS3` / `ANSAWS` tags) record the decision
points: pool evictions, discarded connections, the permanent/transient
classification with the S3 error code, deadline expiry, and background
reconnect attempts.