ANSAWSS3: recover from dead sockets, and report why an upload failed
Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.
Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
pooling it (23 sites). An HTTP error status is not a transport failure - the
server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
the configured one, 5 s by default); a fresh connection keeps the full one.
A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
once at startup, so returning left the handle dead for the process lifetime.
Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
completing just after SetAuthentication cleared the pool cannot leave an
unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
query params, and the gzip Content-Encoding set by UploadTextData. Not
ClearAllHeaders - that wipes SigV4 state (fcc985e).
Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.
Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.
Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# ANSAWSS3 — return codes and retry behaviour
|
||||
|
||||
Reference for callers of the `*ANSAWSHandle` exports in `ANSUtilities.dll`
|
||||
(LabVIEW, C++, .NET). Source of truth: `modules/ANSUtilities/ANSUtilities.h`
|
||||
(`kCall*` constants) and `modules/ANSUtilities/dllmain.cpp`.
|
||||
|
||||
## The one rule
|
||||
|
||||
**`rc == 1` means success. Everything else is a failure.** Success is the only
|
||||
positive value, so `rc == 1` and `rc != 1` are both always correct tests. The
|
||||
negative values exist so a caller can tell *why* it failed, and therefore
|
||||
whether retrying is worthwhile.
|
||||
|
||||
## Upload functions
|
||||
|
||||
Applies to every `Upload*ANSAWSHandle`, `Upload*ANSAWSHandle_CPP` and
|
||||
`Upload*ANSAWSHandle_V2` export (JPEG, binary, text, file-stream, multipart,
|
||||
with or without prefix).
|
||||
|
||||
| Code | Meaning | Retry? |
|
||||
|-----:|---------|--------|
|
||||
| `1` | Uploaded. `uploadedFilePath` holds the object URL. | — |
|
||||
| `0` | A required pointer argument was `NULL`. | **No** — caller bug |
|
||||
| `-1` | Handle is not in the registry: never created, already released, or a stale value. | **No** — re-create the handle |
|
||||
| `-2` | Failed, reason not classified. Should not occur; kept as a fallback. | Yes |
|
||||
| `-3` | A `std::exception` escaped. | Yes |
|
||||
| `-4` | An unknown exception escaped. | Yes |
|
||||
| `-20` | **Permanent.** Bad argument (empty bucket/filename, part size < 5 MB, unreadable file), invalid licence or unlock code, or S3 answered with a 4xx — wrong bucket, no permission, bad key. | **No** — fix the configuration; retrying burns time and changes nothing |
|
||||
| `-21` | **Not connected.** `Connect()` has not succeeded yet, or the link went down and the background retry thread is still working on it. | **Yes, later** — back off a few seconds |
|
||||
| `-22` | **Transient.** Network error, timeout, or HTTP 5xx, still failing after the DLL's own internal retries. | **Yes** |
|
||||
|
||||
A practical LabVIEW policy:
|
||||
|
||||
* `rc = 1` → done.
|
||||
* `rc = -20` or `0` → log it and stop; it will never succeed as-is.
|
||||
* `rc = -21` → queue the image and try again in a few seconds.
|
||||
* anything else negative → retry, with backoff.
|
||||
|
||||
Note that `-22` already means the DLL retried internally (up to 20 attempts
|
||||
within a 30 s budget) before giving up, so an outer retry should not be tight.
|
||||
|
||||
## Connect
|
||||
|
||||
`ConnectANSAWSHandle` / `_V2`:
|
||||
|
||||
| Code | Meaning |
|
||||
|-----:|---------|
|
||||
| `1` | Connected; ready for uploads once `SetAuthentication` is called. |
|
||||
| `0` | A required argument was `NULL`, the handle is unknown, the licence is invalid, or the endpoint resolves but refuses the connection. |
|
||||
| `2` | The endpoint could not be resolved at all (no internet). A background thread keeps retrying — 3 s, doubling to a 30 s ceiling — until it connects or the handle is released. **You do not need to call `Connect` again.** Uploads return `-21` in the meantime. |
|
||||
|
||||
## Other functions
|
||||
|
||||
`CreateANSAWSHandle`, `ReleaseANSAWSHandle`, `SetAuthenticationANSAWSHandle`,
|
||||
`SetProxyANSAWSHandle`, the `ListBucket*` / `CreateBucket*` / `DeleteBucket*`
|
||||
family, `DownloadFileStreamANSAWSHandle` and `GeneratePresignedURLANSAWSHandle`
|
||||
return `1` on success and `0` on failure. `GeneratePresignedURLANSAWSHandle`
|
||||
(the `LStrHandle` variant) additionally uses `-1`, `-2`, `-3`, `-4` with the
|
||||
same meanings as the table above.
|
||||
|
||||
## How connections behave
|
||||
|
||||
* `SetAuthentication` opens and keeps 8 pooled HTTPS connections; uploads borrow
|
||||
and return them, so the TLS handshake is not repeated per image.
|
||||
* A connection whose **transport** failed is closed, never returned to the pool.
|
||||
A connection that merely received an HTTP error status is healthy and is kept.
|
||||
* A pooled connection idle longer than 15 s is closed and replaced rather than
|
||||
handed out, since middleboxes silently drop idle flows.
|
||||
* The first request on a reused connection uses a short idle timeout (a quarter
|
||||
of the configured one, 5 s by default); a fresh connection gets the full
|
||||
timeout. This bounds what a dead socket costs.
|
||||
* One upload call is capped at 30 s of wall clock across all internal retries.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
`ANSAWSS3.log` and DebugView (`AWSS3` / `ANSAWS` tags) record the decision
|
||||
points: pool evictions, discarded connections, the permanent/transient
|
||||
classification with the S3 error code, deadline expiry, and background
|
||||
reconnect attempts.
|
||||
Reference in New Issue
Block a user