ANSAWSS3: recover from dead sockets, and report why an upload failed
Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.
Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
pooling it (23 sites). An HTTP error status is not a transport failure - the
server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
the configured one, 5 s by default); a fresh connection keeps the full one.
A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
once at startup, so returning left the handle dead for the process lifetime.
Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
completing just after SetAuthentication cleared the pool cannot leave an
unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
query params, and the gzip Content-Encoding set by UploadTextData. Not
ClearAllHeaders - that wipes SigV4 state (fcc985e).
Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.
Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.
Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,6 +17,7 @@
|
||||
#include <opencv2/opencv.hpp>
|
||||
#include <thread>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
|
||||
namespace ANSCENTER {
|
||||
@@ -214,6 +215,19 @@ namespace ANSCENTER {
|
||||
CkRest rest;
|
||||
CkSocket socket;
|
||||
CkAuthAws authAws;
|
||||
|
||||
// When this connection last completed a request successfully. A socket
|
||||
// that a NAT/firewall silently killed while idle costs a full
|
||||
// IdleTimeoutMs to discover, so anything idle longer than
|
||||
// kMaxPooledIdleMs is dropped rather than handed out again.
|
||||
std::chrono::steady_clock::time_point lastUsed{ std::chrono::steady_clock::now() };
|
||||
|
||||
// Value of ANSAWSS3::_configGen when this connection was built.
|
||||
// A mismatch means it predates a Connect / SetAuthentication /
|
||||
// SetServerProxy call — e.g. the background RetryLoop finished a
|
||||
// connect just after SetAuthentication cleared the pool — so its
|
||||
// endpoint, proxy or SigV4 state is stale and it must not be reused.
|
||||
uint64_t configGen{ 0 };
|
||||
};
|
||||
|
||||
class ANSULT_API ANSAWSS3 {
|
||||
@@ -252,6 +266,24 @@ namespace ANSCENTER {
|
||||
std::atomic<int> _connectTimeoutMs{ 10000 }; // 10 s
|
||||
std::atomic<int> _idleTimeoutMs { 20000 }; // 20 s
|
||||
|
||||
// Shorter idle timeout used for the FIRST request on a connection
|
||||
// taken from the pool. A pooled socket may have been silently
|
||||
// dropped by a middlebox since it was last used; without this the
|
||||
// caller pays the full _idleTimeoutMs (20 s by default) just to
|
||||
// discover that. A fresh connection still gets the full
|
||||
// _idleTimeoutMs, so the retry after a stale-socket failure is not
|
||||
// penalised. Derived from _idleTimeoutMs by SetTimeouts().
|
||||
std::atomic<int> _reusedIdleTimeoutMs{ 5000 }; // 5 s
|
||||
|
||||
// Bumped by Connect / SetAuthentication / SetServerProxy. Stamped
|
||||
// into every S3Connection at creation; see S3Connection::configGen.
|
||||
std::atomic<uint64_t> _configGen{ 1 };
|
||||
|
||||
// A pooled connection idle longer than this is closed instead of
|
||||
// reused. Kept below the ~20 s after which S3 (and most NAT
|
||||
// devices) stop caring about an idle keep-alive flow.
|
||||
static constexpr int kMaxPooledIdleMs = 15000;
|
||||
|
||||
// Proxy settings
|
||||
std::string _proxyHost;
|
||||
int _proxyPort{ 0 };
|
||||
@@ -265,7 +297,14 @@ namespace ANSCENTER {
|
||||
// Used by TryConnect's first attempt, which is allowed to fail for path-style/MinIO servers.
|
||||
std::unique_ptr<S3Connection> CreateConnection(bool logOnFailure = true);
|
||||
std::unique_ptr<S3Connection> AcquireConnection();
|
||||
// Return a healthy connection to the pool for reuse.
|
||||
void ReleaseConnection(std::unique_ptr<S3Connection> conn);
|
||||
// Close and destroy a connection instead of pooling it. Use on any
|
||||
// TRANSPORT failure (send/recv failed, timeout): that socket is
|
||||
// suspect, and putting it back means the next worker inherits the
|
||||
// same stall. An HTTP error status is NOT a transport failure — the
|
||||
// server answered — so those still go through ReleaseConnection.
|
||||
void DiscardConnection(std::unique_ptr<S3Connection> conn);
|
||||
|
||||
std::string ExtractFileName(const std::string& filePath);
|
||||
std::string GetContentType(const std::string& filePath);
|
||||
@@ -279,8 +318,34 @@ namespace ANSCENTER {
|
||||
// The lambda should write a short human-readable reason into
|
||||
// `lastError` on any non-Success outcome so it can be logged.
|
||||
enum class AttemptResult { Success, Transient, Permanent };
|
||||
// Record the outcome of the current call for the calling thread.
|
||||
static void SetCallStatus(int status);
|
||||
// Shared licence / unlock / connected check for the upload entry
|
||||
// points. Logs the reason and sets the per-thread call status
|
||||
// (permanent for a licence problem, not-connected for a link that
|
||||
// has not come up yet). Returns false if the call must not proceed.
|
||||
bool CheckUploadPreconditions(const std::string& opName);
|
||||
|
||||
// Decide what a failed Chilkat request means, and dispose of the
|
||||
// connection accordingly. Chilkat reports failure for an HTTP 4xx
|
||||
// delivered during an Expect: 100-continue exchange, so "the call
|
||||
// returned false" does NOT imply the network failed — a 404
|
||||
// NoSuchBucket arrives this way. Retrying those wastes the whole
|
||||
// retry budget and reports the wrong class to the caller.
|
||||
// 4xx (except 408/429) -> Permanent, socket healthy -> pooled
|
||||
// anything else -> Transient, socket suspect -> discarded
|
||||
// Takes ownership of `conn`, and rewrites `lastError` into one
|
||||
// readable line.
|
||||
AttemptResult ClassifyFailedRequest(std::unique_ptr<S3Connection> conn,
|
||||
std::string& lastError);
|
||||
static constexpr int kUploadMaxAttempts = 20;
|
||||
static constexpr int kUploadRetryDelayMs = 100;
|
||||
// Wall-clock budget for one upload call, across all attempts. The
|
||||
// attempt cap alone is not a bound on time: 20 attempts that each
|
||||
// hit a 20 s idle timeout would block the caller for ~400 s. A new
|
||||
// attempt is only started while inside this budget, so the worst
|
||||
// case is the budget plus one in-flight attempt.
|
||||
static constexpr int kUploadDeadlineMs = 30000;
|
||||
bool UploadWithRetry(
|
||||
const std::string& opName,
|
||||
const std::function<AttemptResult(std::string& lastError)>& attemptFn);
|
||||
@@ -294,6 +359,37 @@ namespace ANSCENTER {
|
||||
bool TryConnect(bool& awsPath);
|
||||
|
||||
public:
|
||||
// ---- Call status codes -------------------------------------------
|
||||
// Every upload entry point records one of these for the CALLING
|
||||
// THREAD, so the C exports can tell the caller whether a retry is
|
||||
// worth attempting without changing any function signature.
|
||||
//
|
||||
// These EXTEND the codes the C wrappers already return; they do not
|
||||
// redefine any of them. Full set seen by a LabVIEW caller:
|
||||
// 1 success
|
||||
// 0 null / invalid argument passed to the export
|
||||
// -1 handle not in the registry
|
||||
// -2 upload failed, reason not classified (fallback)
|
||||
// -3 std::exception escaped
|
||||
// -4 unknown exception escaped
|
||||
// -20 permanent: bad argument, licence, bucket, or HTTP 4xx
|
||||
// -21 not connected yet / reconnect in progress
|
||||
// -22 transient: network, timeout or 5xx after internal retries
|
||||
//
|
||||
// Retry rule for the caller: retry on -21 and -22 (and -2/-3/-4 if
|
||||
// it wants to be generous); do NOT retry on -20 or 0 — those need a
|
||||
// configuration or code fix. Success stays the only positive value,
|
||||
// so both `rc == 1` and `rc != 1` remain correct tests.
|
||||
static constexpr int kCallSuccess = 1;
|
||||
static constexpr int kCallPermanentError = -20;
|
||||
static constexpr int kCallNotConnected = -21;
|
||||
static constexpr int kCallTransientError = -22;
|
||||
|
||||
// Status of the most recent upload call made ON THIS THREAD.
|
||||
// Thread-local, so concurrent workers sharing one handle never see
|
||||
// each other's result. Read it immediately after the call.
|
||||
static int LastCallStatus();
|
||||
|
||||
ANSAWSS3();
|
||||
~ANSAWSS3();
|
||||
[[nodiscard]] bool Initialize(const std::string& licenseKey);
|
||||
|
||||
Reference in New Issue
Block a user