ANSAWSS3: recover from dead sockets, and report why an upload failed

Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 21:24:12 +10:00
co-authored by Claude Opus 5
parent 6f1d069d14
commit 86b520d85c
4 changed files with 606 additions and 206 deletions
+96
View File
@@ -17,6 +17,7 @@
#include <opencv2/opencv.hpp>
#include <thread>
#include <atomic>
#include <chrono>
#include <functional>
namespace ANSCENTER {
@@ -214,6 +215,19 @@ namespace ANSCENTER {
CkRest rest;
CkSocket socket;
CkAuthAws authAws;
// When this connection last completed a request successfully. A socket
// that a NAT/firewall silently killed while idle costs a full
// IdleTimeoutMs to discover, so anything idle longer than
// kMaxPooledIdleMs is dropped rather than handed out again.
std::chrono::steady_clock::time_point lastUsed{ std::chrono::steady_clock::now() };
// Value of ANSAWSS3::_configGen when this connection was built.
// A mismatch means it predates a Connect / SetAuthentication /
// SetServerProxy call — e.g. the background RetryLoop finished a
// connect just after SetAuthentication cleared the pool — so its
// endpoint, proxy or SigV4 state is stale and it must not be reused.
uint64_t configGen{ 0 };
};
class ANSULT_API ANSAWSS3 {
@@ -252,6 +266,24 @@ namespace ANSCENTER {
std::atomic<int> _connectTimeoutMs{ 10000 }; // 10 s
std::atomic<int> _idleTimeoutMs { 20000 }; // 20 s
// Shorter idle timeout used for the FIRST request on a connection
// taken from the pool. A pooled socket may have been silently
// dropped by a middlebox since it was last used; without this the
// caller pays the full _idleTimeoutMs (20 s by default) just to
// discover that. A fresh connection still gets the full
// _idleTimeoutMs, so the retry after a stale-socket failure is not
// penalised. Derived from _idleTimeoutMs by SetTimeouts().
std::atomic<int> _reusedIdleTimeoutMs{ 5000 }; // 5 s
// Bumped by Connect / SetAuthentication / SetServerProxy. Stamped
// into every S3Connection at creation; see S3Connection::configGen.
std::atomic<uint64_t> _configGen{ 1 };
// A pooled connection idle longer than this is closed instead of
// reused. Kept below the ~20 s after which S3 (and most NAT
// devices) stop caring about an idle keep-alive flow.
static constexpr int kMaxPooledIdleMs = 15000;
// Proxy settings
std::string _proxyHost;
int _proxyPort{ 0 };
@@ -265,7 +297,14 @@ namespace ANSCENTER {
// Used by TryConnect's first attempt, which is allowed to fail for path-style/MinIO servers.
std::unique_ptr<S3Connection> CreateConnection(bool logOnFailure = true);
std::unique_ptr<S3Connection> AcquireConnection();
// Return a healthy connection to the pool for reuse.
void ReleaseConnection(std::unique_ptr<S3Connection> conn);
// Close and destroy a connection instead of pooling it. Use on any
// TRANSPORT failure (send/recv failed, timeout): that socket is
// suspect, and putting it back means the next worker inherits the
// same stall. An HTTP error status is NOT a transport failure — the
// server answered — so those still go through ReleaseConnection.
void DiscardConnection(std::unique_ptr<S3Connection> conn);
std::string ExtractFileName(const std::string& filePath);
std::string GetContentType(const std::string& filePath);
@@ -279,8 +318,34 @@ namespace ANSCENTER {
// The lambda should write a short human-readable reason into
// `lastError` on any non-Success outcome so it can be logged.
enum class AttemptResult { Success, Transient, Permanent };
// Record the outcome of the current call for the calling thread.
static void SetCallStatus(int status);
// Shared licence / unlock / connected check for the upload entry
// points. Logs the reason and sets the per-thread call status
// (permanent for a licence problem, not-connected for a link that
// has not come up yet). Returns false if the call must not proceed.
bool CheckUploadPreconditions(const std::string& opName);
// Decide what a failed Chilkat request means, and dispose of the
// connection accordingly. Chilkat reports failure for an HTTP 4xx
// delivered during an Expect: 100-continue exchange, so "the call
// returned false" does NOT imply the network failed — a 404
// NoSuchBucket arrives this way. Retrying those wastes the whole
// retry budget and reports the wrong class to the caller.
// 4xx (except 408/429) -> Permanent, socket healthy -> pooled
// anything else -> Transient, socket suspect -> discarded
// Takes ownership of `conn`, and rewrites `lastError` into one
// readable line.
AttemptResult ClassifyFailedRequest(std::unique_ptr<S3Connection> conn,
std::string& lastError);
static constexpr int kUploadMaxAttempts = 20;
static constexpr int kUploadRetryDelayMs = 100;
// Wall-clock budget for one upload call, across all attempts. The
// attempt cap alone is not a bound on time: 20 attempts that each
// hit a 20 s idle timeout would block the caller for ~400 s. A new
// attempt is only started while inside this budget, so the worst
// case is the budget plus one in-flight attempt.
static constexpr int kUploadDeadlineMs = 30000;
bool UploadWithRetry(
const std::string& opName,
const std::function<AttemptResult(std::string& lastError)>& attemptFn);
@@ -294,6 +359,37 @@ namespace ANSCENTER {
bool TryConnect(bool& awsPath);
public:
// ---- Call status codes -------------------------------------------
// Every upload entry point records one of these for the CALLING
// THREAD, so the C exports can tell the caller whether a retry is
// worth attempting without changing any function signature.
//
// These EXTEND the codes the C wrappers already return; they do not
// redefine any of them. Full set seen by a LabVIEW caller:
// 1 success
// 0 null / invalid argument passed to the export
// -1 handle not in the registry
// -2 upload failed, reason not classified (fallback)
// -3 std::exception escaped
// -4 unknown exception escaped
// -20 permanent: bad argument, licence, bucket, or HTTP 4xx
// -21 not connected yet / reconnect in progress
// -22 transient: network, timeout or 5xx after internal retries
//
// Retry rule for the caller: retry on -21 and -22 (and -2/-3/-4 if
// it wants to be generous); do NOT retry on -20 or 0 — those need a
// configuration or code fix. Success stays the only positive value,
// so both `rc == 1` and `rc != 1` remain correct tests.
static constexpr int kCallSuccess = 1;
static constexpr int kCallPermanentError = -20;
static constexpr int kCallNotConnected = -21;
static constexpr int kCallTransientError = -22;
// Status of the most recent upload call made ON THIS THREAD.
// Thread-local, so concurrent workers sharing one handle never see
// each other's result. Read it immediately after the call.
static int LastCallStatus();
ANSAWSS3();
~ANSAWSS3();
[[nodiscard]] bool Initialize(const std::string& licenseKey);