ANSAWSS3: recover from dead sockets, and report why an upload failed

Field logs showed clusters of uploads taking exactly 20004/20007/20003 ms —
the default _idleTimeoutMs — across several workers at the same instant, then
recurring. Cause: a connection whose request failed was returned to the pool,
so a socket silently dropped by a middlebox was handed to worker after worker,
each paying the full idle timeout to rediscover it. Chilkat's autoReconnect
only covers a cleanly closed connection, not a half-open one.

Connection handling:
- DiscardConnection(): a TRANSPORT failure closes the connection instead of
  pooling it (23 sites). An HTTP error status is not a transport failure - the
  server answered - so those connections are still reused.
- Pooled connections idle longer than 15 s are dropped at acquire time.
- First request on a reused connection uses a short idle timeout (a quarter of
  the configured one, 5 s by default); a fresh connection keeps the full one.
  A dead socket now costs ~5 s once, not 20 s repeatedly.
- One upload call is capped at 30 s wall-clock. The attempt count alone was not
  a time bound: 20 attempts x 20 s could block the caller for ~400 s.
- RetryLoop no longer gives up after one failed connect. Callers call Connect()
  once at startup, so returning left the handle dead for the process lifetime.
  Now backs off 3 s -> 30 s until connected or stopped.
- Connections carry the config generation they were built against, so a connect
  completing just after SetAuthentication cleared the pool cannot leave an
  unauthenticated connection behind (those uploads would 403).
- Stale per-request state is cleared on acquire: multipart uploadId/partNumber
  query params, and the gzip Content-Encoding set by UploadTextData. Not
  ClearAllHeaders - that wipes SigV4 state (fcc985e).

Return codes - upload exports now say whether a retry is worthwhile:
-20 permanent (bad argument, licence, or HTTP 4xx), -21 not connected yet,
-22 transient. Recorded per thread, since one handle is shared by many workers.
Success stays the only positive value, so rc == 1 and rc != 1 are unchanged.
Handle and exception codes (-1/-3/-4) made consistent across all upload
wrappers; some previously returned 0 for these.

Also fixes a misclassification this surfaced: Chilkat reports failure for an
HTTP 4xx delivered during an Expect: 100-continue exchange, so NoSuchBucket was
treated as a network error and retried 20 times before reporting "transient".
The failure path now checks the response status. Chilkat's ~2 KB error dump per
failure is compressed to one line carrying the actual S3 error code.

Verified against live S3: 14,701 uploads across 15 threads with a 150 s idle
gap, 0 failures, no call over 5 s (avg 90 ms, p95 136 ms), matching the
pre-change baseline; and all six return-code classes confirmed end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 21:24:12 +10:00
co-authored by Claude Opus 5
parent 6f1d069d14
commit 86b520d85c
4 changed files with 606 additions and 206 deletions
+332 -122
View File
@@ -168,6 +168,78 @@ namespace ANSCENTER
return out.str();
}
// ── Per-thread call status ───────────────────────────────────────────────
// Written by every upload entry point, read by the C exports in dllmain so
// the caller learns WHY a call failed (and therefore whether to retry)
// without any change to the exported signatures. Thread-local because a
// single ANSAWSS3 handle is routinely shared by many worker threads — a
// per-instance field would report another worker's outcome.
static thread_local int t_callStatus = ANSAWSS3::kCallPermanentError;
void ANSAWSS3::SetCallStatus(int status) { t_callStatus = status; }
int ANSAWSS3::LastCallStatus() { return t_callStatus; }
// ────────────────────────────────────────────────────────────────────────
// SummarizeChilkatRequestError: one line out of a request-level Chilkat log.
//
// The raw log is ~2 KB of call-stack scaffolding per failure. Written once
// per retry, per worker, it buries the one thing an operator needs: what S3
// actually said. S3 returns its reason as XML in the response body, which
// Chilkat echoes into the log:
// <Error><Code>NoSuchBucket</Code><Message>The specified bucket does
// not exist</Message>...
// Pull that out, prefix the status code, and drop the rest.
// ────────────────────────────────────────────────────────────────────────
static std::string SummarizeChilkatRequestError(const char* raw, int statusCode) {
std::ostringstream out;
if (statusCode > 0) out << "HTTP " << statusCode;
auto tagValue = [](const std::string& s, const char* tag) -> std::string {
const std::string open = std::string("<") + tag + ">";
const std::string close = std::string("</") + tag + ">";
const size_t a = s.find(open);
if (a == std::string::npos) return {};
const size_t b = s.find(close, a + open.size());
if (b == std::string::npos) return {};
return s.substr(a + open.size(), b - a - open.size());
};
if (raw && *raw) {
const std::string s(raw);
const std::string code = tagValue(s, "Code");
const std::string msg = tagValue(s, "Message");
if (!code.empty()) {
if (statusCode > 0) out << " ";
out << code;
if (!msg.empty()) out << ": " << msg;
}
else {
// No S3 error document — fall back to the deepest concrete line
// (same idea as SummarizeChilkatError, without the connect
// specific fields).
std::istringstream iss(s);
std::string ln, leaf;
while (std::getline(iss, ln)) {
const size_t a = ln.find_first_not_of(" \t\r\n");
if (a == std::string::npos) continue;
const size_t b = ln.find_last_not_of(" \t\r\n");
ln = ln.substr(a, b - a + 1);
if (ln.rfind("--", 0) == 0) continue; // close marker
if (!ln.empty() && ln.back() == ':') continue; // stack label
if (ln == "Failed.") continue;
leaf = ln;
}
if (!leaf.empty()) {
if (statusCode > 0) out << " — ";
out << leaf;
}
}
}
std::string result = out.str();
return result.empty() ? "(no detail in Chilkat log)" : result;
}
// Private helper function to extract file name from a path
// Helper function to extract filename from path
std::string ANSAWSS3::ExtractFileName(const std::string& filePath) {
@@ -302,12 +374,23 @@ namespace ANSCENTER
return true;
}
// Background connect loop, started by Connect() when the endpoint could not
// be reached at all (return code 2).
//
// This loop NEVER gives up on its own. The caller — a LabVIEW VI, an
// unattended AIBOX — calls Connect() exactly once at startup, so if this
// thread returned while still unconnected the handle would stay dead for
// the life of the process and every upload would fail instantly. Backoff
// grows 3 s -> 30 s so a long outage costs little, and the loop only exits
// once connected or when StopRetry() is called.
void ANSAWSS3::RetryLoop() {
int attempt = 0;
int backoffSec = 3;
constexpr int kMaxBackoffSec = 30;
while (!_stopRetry) {
// Wait 3 seconds (in 1-second increments so we can respond to stop quickly)
for (int i = 0; i < 3 && !_stopRetry; ++i) {
// Sleep in 1-second increments so we can respond to stop quickly.
for (int i = 0; i < backoffSec && !_stopRetry; ++i) {
std::this_thread::sleep_for(std::chrono::seconds(1));
}
if (_stopRetry) break;
@@ -327,8 +410,12 @@ namespace ANSCENTER
}
if (!resolved) {
_logger.LogDebug("ANSAWSS3::RetryLoop",
"Retry #" + std::to_string(attempt) + " - no internet, retrying in 3s",
"Retry #" + std::to_string(attempt) + " - no internet, retrying in " +
std::to_string(backoffSec) + "s",
__FILE__, __LINE__);
ANS_DBG("AWSS3", "RetryLoop #%d: DNS still failing, next try in %ds",
attempt, backoffSec);
backoffSec = (std::min)(kMaxBackoffSec, backoffSec * 2);
continue;
}
}
@@ -341,19 +428,24 @@ namespace ANSCENTER
_logger.LogDebug("ANSAWSS3::RetryLoop",
"Connected successfully after " + std::to_string(attempt) + " retries",
__FILE__, __LINE__);
_retryInProgress = false;
return;
}
else {
// Internet is available but connection failed (bad auth/URL)
// Stop retrying — no point, the parameters are wrong
_logger.LogError("ANSAWSS3::RetryLoop",
"Internet available but connection failed (check authentication or URL). Stopping retry.",
__FILE__, __LINE__);
ANS_DBG("AWSS3", "RetryLoop: connected after %d attempt(s)", attempt);
_retryInProgress = false;
return;
}
}
// DNS resolves but the connection failed — bad credentials, wrong
// URL, or the endpoint is refusing us. Keep trying anyway: the
// caller has no other way back, and this state is often temporary
// (endpoint rebooting, TLS interception clearing, clock skew being
// corrected). Backoff keeps the cost negligible.
_logger.LogError("ANSAWSS3::RetryLoop",
"Retry #" + std::to_string(attempt) +
" - endpoint resolves but connection failed (check authentication or URL). "
"Retrying in " + std::to_string(backoffSec) + "s",
__FILE__, __LINE__);
ANS_DBG("AWSS3", "RetryLoop #%d: connect failed, next try in %ds", attempt, backoffSec);
backoffSec = (std::min)(kMaxBackoffSec, backoffSec * 2);
}
_retryInProgress = false;
}
@@ -366,9 +458,16 @@ namespace ANSCENTER
std::unique_ptr<S3Connection> ANSAWSS3::CreateConnection(bool logOnFailure) {
auto conn = std::make_unique<S3Connection>();
// Stamp the config generation this connection is built against so a
// later Connect / SetAuthentication / SetServerProxy can invalidate it
// even if it lands in the pool after that call cleared it.
conn->configGen = _configGen.load();
// Apply request timeouts BEFORE Connect() so ConnectTimeoutMs takes
// effect on this very handshake. IdleTimeoutMs is sticky and applies
// to every subsequent request on this CkRest.
// to every subsequent request on this CkRest. A brand-new connection
// gets the FULL idle timeout — it is known-good, so it should be given
// every chance to complete; only reuse gets the short leash.
conn->rest.put_ConnectTimeoutMs(_connectTimeoutMs.load());
conn->rest.put_IdleTimeoutMs(_idleTimeoutMs.load());
@@ -423,26 +522,82 @@ namespace ANSCENTER
return conn;
}
// Take a connection from the pool, skipping any that are stale (idle too
// long) or built against superseded config. Falls through to a fresh
// connection when nothing reusable is left.
std::unique_ptr<S3Connection> ANSAWSS3::AcquireConnection() {
{
std::lock_guard<std::mutex> lk(_poolMutex);
if (!_pool.empty()) {
auto conn = std::move(_pool.back());
const uint64_t gen = _configGen.load();
const auto now = std::chrono::steady_clock::now();
for (;;) {
std::unique_ptr<S3Connection> conn;
{
std::lock_guard<std::mutex> lk(_poolMutex);
if (_pool.empty()) break;
conn = std::move(_pool.back());
_pool.pop_back();
return conn;
}
if (conn->configGen != gen) {
ANS_DBG("AWSS3", "Acquire: dropping pooled conn from older config gen %llu (now %llu)",
(unsigned long long)conn->configGen, (unsigned long long)gen);
DiscardConnection(std::move(conn)); // pool lock released first
continue;
}
const long long idleMs = std::chrono::duration_cast<std::chrono::milliseconds>(
now - conn->lastUsed).count();
if (idleMs > kMaxPooledIdleMs) {
ANS_DBG("AWSS3", "Acquire: dropping pooled conn idle for %lldms (> %dms)",
idleMs, kMaxPooledIdleMs);
DiscardConnection(std::move(conn));
continue;
}
// Clear per-request state left behind by whoever used this
// connection last. Multipart uploads set uploadId / partNumber
// query params; a plain PUT that inherited them would target the
// wrong resource. Content-Encoding is set by UploadTextData only,
// and a JPEG that inherited "gzip" would be stored with metadata
// saying it is compressed — browsers would then fail to display it.
// NOTE: do NOT call ClearAllHeaders here — it wipes Chilkat's SigV4
// state (see commit fcc985e). Headers that every upload sets
// explicitly (Content-Type, Expect) are overwritten by AddHeader
// and need no clearing.
conn->rest.ClearAllQueryParams();
conn->rest.RemoveHeader("Content-Encoding");
// Short leash on reuse: if this socket died while idle, fail fast
// and let the retry run on a fresh connection.
conn->rest.put_IdleTimeoutMs(_reusedIdleTimeoutMs.load());
return conn;
}
// Pool empty — create a new connection (no lock held during network I/O)
// Pool empty (or everything in it was stale) — create a new connection.
std::lock_guard<std::mutex> cfgLk(_configMutex);
return CreateConnection();
}
void ANSAWSS3::ReleaseConnection(std::unique_ptr<S3Connection> conn) {
if (!conn) return;
// Mark as fresh and restore the full idle timeout; AcquireConnection
// re-applies the short one when it hands the connection out again.
conn->lastUsed = std::chrono::steady_clock::now();
conn->rest.put_IdleTimeoutMs(_idleTimeoutMs.load());
std::lock_guard<std::mutex> lk(_poolMutex);
_pool.push_back(std::move(conn));
}
// Close a suspect connection instead of pooling it. Disconnect is given a
// small budget only — the point is to release the socket, not to wait on a
// peer that has already stopped answering.
void ANSAWSS3::DiscardConnection(std::unique_ptr<S3Connection> conn) {
if (!conn) return;
conn->rest.Disconnect(100);
ANS_DBG("AWSS3", "Discarded connection (not returned to pool)");
// unique_ptr destructor tears the CkRest down here.
}
// Pre-populate the pool with `count` ready-to-use connections. Each
// connection performs a TLS handshake; doing `count` of them in parallel
// (rather than serially as AcquireConnection would on a cold pool) cuts
@@ -509,6 +664,16 @@ namespace ANSCENTER
_connectTimeoutMs.store(connectMs);
_idleTimeoutMs.store(idleMs);
// Reuse timeout scales with the configured idle timeout (a quarter of
// it, clamped to a sane floor) so a deployment on a slow link that
// raises idleMs does not end up with an over-aggressive leash on
// pooled sockets.
// Never longer than the configured idle timeout itself (idleMs can be
// as low as 1000), and never so short that a healthy but busy server
// gets cut off.
const int reusedMs = (std::min)(idleMs, (std::max)(2000, idleMs / 4));
_reusedIdleTimeoutMs.store(reusedMs);
// Propagate to existing pooled connections so already-warm workers
// start using the new values on their next request.
{
@@ -522,16 +687,68 @@ namespace ANSCENTER
}
_logger.LogDebug("ANSAWSS3::SetTimeouts",
"connect=" + std::to_string(connectMs) + "ms, idle=" + std::to_string(idleMs) + "ms",
"connect=" + std::to_string(connectMs) + "ms, idle=" + std::to_string(idleMs) +
"ms, idle-on-reuse=" + std::to_string(reusedMs) + "ms",
__FILE__, __LINE__);
return true;
}
// Shared precondition check for every upload entry point. The distinction
// matters to the caller: a licence problem needs an operator, whereas "not
// connected" clears itself once the background RetryLoop gets through — so
// the two report different status codes.
bool ANSAWSS3::CheckUploadPreconditions(const std::string& opName) {
if (!_isLicenseValid || !_isUnlockCodeValid) {
_logger.LogError(opName,
!_isLicenseValid ? "Invalid license" : "Invalid unlock code",
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError);
return false;
}
if (!_bConnected) {
const bool retrying = _retryInProgress.load();
_logger.LogError(opName,
retrying ? "Not connected (waiting for internet, retrying in background)"
: "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
ANS_DBG("AWSS3", "%s: not connected (background retry %s) — caller should retry later",
opName.c_str(), retrying ? "active" : "inactive");
SetCallStatus(kCallNotConnected);
return false;
}
return true;
}
// See the declaration comment in ANSUtilities.h for why a failed Chilkat
// request is not automatically a network problem.
ANSAWSS3::AttemptResult ANSAWSS3::ClassifyFailedRequest(
std::unique_ptr<S3Connection> conn, std::string& lastError) {
const int statusCode = conn ? conn->rest.get_ResponseStatusCode() : 0;
lastError = SummarizeChilkatRequestError(
conn ? conn->rest.lastErrorText() : nullptr, statusCode);
// The server answered, and its answer will not change on a retry:
// wrong bucket, no permission, bad key. 408 and 429 are the exceptions
// — those explicitly ask the client to try again.
if (statusCode >= 400 && statusCode <= 499 &&
statusCode != 408 && statusCode != 429) {
ANS_DBG("AWSS3", "Request failed permanently: %s", lastError.c_str());
ReleaseConnection(std::move(conn)); // the socket itself is fine
return AttemptResult::Permanent;
}
ANS_DBG("AWSS3", "Request failed transiently: %s", lastError.c_str());
DiscardConnection(std::move(conn)); // suspect socket — do not pool
return AttemptResult::Transient;
}
// Drives a retry loop for a single upload operation. See the declaration
// comment in ANSUtilities.h for the AttemptResult contract.
//
// Retry policy (shared across all upload entry points):
// - kUploadMaxAttempts attempts total
// - kUploadMaxAttempts attempts total, AND kUploadDeadlineMs wall-clock
// (whichever comes first — the attempt count alone does not bound how
// long the caller blocks, since one attempt can sit on a timeout)
// - kUploadRetryDelayMs between attempts (no delay after the last one)
// - std::exception thrown inside attemptFn is treated as Transient
// - Any other exception is treated as Permanent (bail immediately)
@@ -540,6 +757,11 @@ namespace ANSCENTER
const std::function<AttemptResult(std::string& lastError)>& attemptFn) {
std::string lastError;
const auto started = std::chrono::steady_clock::now();
auto elapsedMs = [&started]() {
return std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now() - started).count();
};
for (int attempt = 1; attempt <= kUploadMaxAttempts; ++attempt) {
AttemptResult result = AttemptResult::Transient;
@@ -556,6 +778,7 @@ namespace ANSCENTER
}
if (result == AttemptResult::Success) {
SetCallStatus(kCallSuccess);
return true;
}
@@ -563,14 +786,29 @@ namespace ANSCENTER
_logger.LogError(opName,
"Permanent failure (no retry): " + lastError,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError);
return false;
}
// Transient — retry if both budgets allow it.
if (elapsedMs() >= kUploadDeadlineMs) {
_logger.LogError(opName,
"Upload gave up after " + std::to_string(attempt) + " attempt(s) / " +
std::to_string(elapsedMs()) + "ms (deadline " +
std::to_string(kUploadDeadlineMs) + "ms). Last error: " + lastError,
__FILE__, __LINE__);
ANS_DBG("AWSS3", "%s: deadline exceeded after %d attempt(s), %lldms",
opName.c_str(), attempt, elapsedMs());
// Ran out of time on transient errors — the caller should retry.
SetCallStatus(kCallTransientError);
return false;
}
// Transient — retry if attempts remain.
if (attempt < kUploadMaxAttempts) {
_logger.LogDebug(opName,
"Attempt " + std::to_string(attempt) + "/" +
std::to_string(kUploadMaxAttempts) + " failed: " + lastError +
std::to_string(kUploadMaxAttempts) + " failed after " +
std::to_string(elapsedMs()) + "ms: " + lastError +
" — retrying in " + std::to_string(kUploadRetryDelayMs) + "ms",
__FILE__, __LINE__);
std::this_thread::sleep_for(
@@ -580,8 +818,10 @@ namespace ANSCENTER
_logger.LogError(opName,
"Upload failed after " + std::to_string(kUploadMaxAttempts) +
" attempts. Last error: " + lastError,
" attempts / " + std::to_string(elapsedMs()) + "ms. Last error: " + lastError,
__FILE__, __LINE__);
// Every attempt hit a transient error — worth retrying from the caller.
SetCallStatus(kCallTransientError);
return false;
}
@@ -652,7 +892,10 @@ namespace ANSCENTER
// Simplified proxy validation logic
_bProxy = !proxyHost.empty() && proxyPort > 0;
// Clear pool so new connections pick up proxy config
// Clear pool so new connections pick up proxy config. Bumping the
// generation also invalidates any connection still in flight
// elsewhere, which would otherwise be pooled after this clear.
_configGen.fetch_add(1);
{
std::lock_guard<std::mutex> lk(_poolMutex);
_pool.clear();
@@ -701,7 +944,9 @@ namespace ANSCENTER
_bTls = bTls;
_bAutoReconnect = autoReconnect;
// Clear pool so new connections use updated config
// Clear pool so new connections use updated config (see the
// generation note in SetServerProxy).
_configGen.fetch_add(1);
{
std::lock_guard<std::mutex> lk(_poolMutex);
_pool.clear();
@@ -770,7 +1015,13 @@ namespace ANSCENTER
_secretKey = secretKey;
_authReady = true;
// Clear pool so new connections pick up new credentials.
// Clear pool so new connections pick up new credentials. The
// generation bump matters most here: the background RetryLoop
// can complete a connect at any moment, and a connection it
// built before _authReady was set carries no SigV4 state — if
// it landed in the pool after this clear, every upload using it
// would come back 403 (a permanent failure, so not retried).
_configGen.fetch_add(1);
std::lock_guard<std::mutex> lk(_poolMutex);
_pool.clear();
}
@@ -809,7 +1060,7 @@ namespace ANSCENTER
if (!conn->rest.FullRequestNoBodySb("GET", "/", sbResponse)) {
_logger.LogError("ANSAWSS3::ListBuckets - Request failed",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return {};
}
@@ -909,7 +1160,7 @@ namespace ANSCENTER
if (!success) {
_logger.LogError("ANSAWSS3::ListBucketObjects - Request failed on page " + std::to_string(pageCount),
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return objectList; // Return what we have so far
}
@@ -1071,7 +1322,7 @@ namespace ANSCENTER
if (!success) {
_logger.LogError("ANSAWSS3::ListBucketObjectsWithPrefix - Request failed on page " + std::to_string(pageCount),
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return objectList; // Return what we have so far
}
@@ -1227,7 +1478,7 @@ namespace ANSCENTER
if (conn->rest.get_LastMethodSuccess() != true) {
_logger.LogError("ANSAWSS3::CreateBucket - Request failed",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return false;
}
@@ -1314,7 +1565,7 @@ namespace ANSCENTER
_logger.LogError("ANSAWSS3::DeleteBucket",
"Failed to set AWS authentication: " + std::string(conn->rest.lastErrorText()),
__FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return false;
}
@@ -1440,7 +1691,7 @@ namespace ANSCENTER
if (conn->rest.get_LastMethodSuccess() != true) {
_logger.LogError("ANSAWSS3::CreateFolder - Request failed",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return false;
}
@@ -1608,7 +1859,7 @@ namespace ANSCENTER
if (conn->rest.get_LastMethodSuccess() != true) {
_logger.LogError("ANSAWSS3::GetBucketRegion - Request failed",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return "";
}
@@ -1679,15 +1930,11 @@ namespace ANSCENTER
// _logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || textFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -1697,6 +1944,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Failed to extract filename from path: " + textFilePath,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
const std::string contentType = GetContentType(textFilePath);
@@ -1740,9 +1988,7 @@ namespace ANSCENTER
conn->rest.fullRequestString("PUT", objectPath.c_str(), fileContents);
if (!conn->rest.get_LastMethodSuccess()) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -1772,15 +2018,11 @@ namespace ANSCENTER
// _logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || dataFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -1789,6 +2031,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Failed to extract filename from path: " + dataFilePath,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
const std::string contentType = GetContentType(dataFilePath);
@@ -1821,9 +2064,7 @@ namespace ANSCENTER
CkStringBuilder sbResponse;
if (!conn->rest.FullRequestBd("PUT", objectPath.c_str(), binData, sbResponse)) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -1852,15 +2093,11 @@ namespace ANSCENTER
//_logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || dataFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -1872,6 +2109,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Failed to extract filename from path: " + dataFilePath,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
}
@@ -1923,9 +2161,7 @@ namespace ANSCENTER
CkStringBuilder sbResponse;
if (!conn->rest.FullRequestBd("PUT", objectPath.c_str(), binData, sbResponse)) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -1956,15 +2192,11 @@ namespace ANSCENTER
//_logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || dataFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -1994,9 +2226,7 @@ namespace ANSCENTER
conn->rest.fullRequestStream("PUT", objectPath.c_str(), fileStream);
if (conn->rest.get_LastMethodSuccess() != true) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -2026,15 +2256,11 @@ namespace ANSCENTER
//_logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || dataFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2042,6 +2268,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Part size must be at least 5MB (5242880 bytes)",
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2049,6 +2276,7 @@ namespace ANSCENTER
CkFileAccess fac;
if (!fac.FileExists(dataFilePath.c_str())) {
_logger.LogError(kOp, "File not found: " + dataFilePath, __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
}
@@ -2058,6 +2286,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Failed to extract filename from path: " + dataFilePath,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
@@ -2106,9 +2335,8 @@ namespace ANSCENTER
const char* responseXml = conn->rest.fullRequestNoBody("POST", objectPath.c_str());
if (!conn->rest.get_LastMethodSuccess()) {
lastError = std::string("Initiate failed: ") + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Initiate: " + lastError; return r; }
}
int initStatus = conn->rest.get_ResponseStatusCode();
if (initStatus != 200) {
@@ -2160,9 +2388,8 @@ namespace ANSCENTER
const char* responseStr = conn->rest.fullRequestStream("PUT", objectPath.c_str(), fileStream);
if (!conn->rest.get_LastMethodSuccess()) {
lastError = "Part " + std::to_string(partNumber) + " failed: " + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Part " + std::to_string(partNumber) + ": " + lastError; return r; }
}
int partStatus = conn->rest.get_ResponseStatusCode();
if (partStatus != 200) {
@@ -2175,7 +2402,7 @@ namespace ANSCENTER
const char* etag = conn->rest.responseHdrByName("ETag");
if (!conn->rest.get_LastMethodSuccess() || etag == nullptr || etag[0] == '\0') {
lastError = "Part " + std::to_string(partNumber) + ": ETag missing from response";
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return AttemptResult::Transient;
}
@@ -2194,9 +2421,8 @@ namespace ANSCENTER
responseXml = conn->rest.fullRequestString("POST", objectPath.c_str(), partsListXml.getXml());
if (!conn->rest.get_LastMethodSuccess()) {
lastError = std::string("Complete failed: ") + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Complete: " + lastError; return r; }
}
int completeStatus = conn->rest.get_ResponseStatusCode();
if (completeStatus != 200) {
@@ -2227,15 +2453,11 @@ namespace ANSCENTER
//_logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || dataFilePath.empty()) {
_logger.LogError(kOp, "Bucket name or file path is empty", __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2243,6 +2465,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Part size must be at least 5MB (5242880 bytes)",
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2250,6 +2473,7 @@ namespace ANSCENTER
CkFileAccess fac;
if (!fac.FileExists(dataFilePath.c_str())) {
_logger.LogError(kOp, "File not found: " + dataFilePath, __FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
}
@@ -2261,6 +2485,7 @@ namespace ANSCENTER
_logger.LogError(kOp,
"Failed to extract filename from path: " + dataFilePath,
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // bad input path — retrying cannot help
return false;
}
}
@@ -2323,9 +2548,8 @@ namespace ANSCENTER
const char* responseXml = conn->rest.fullRequestNoBody("POST", objectPath.c_str());
if (!conn->rest.get_LastMethodSuccess()) {
lastError = std::string("Initiate failed: ") + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Initiate: " + lastError; return r; }
}
int initStatus = conn->rest.get_ResponseStatusCode();
if (initStatus != 200) {
@@ -2377,9 +2601,8 @@ namespace ANSCENTER
const char* responseStr = conn->rest.fullRequestStream("PUT", objectPath.c_str(), fileStream);
if (!conn->rest.get_LastMethodSuccess()) {
lastError = "Part " + std::to_string(partNumber) + " failed: " + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Part " + std::to_string(partNumber) + ": " + lastError; return r; }
}
int partStatus = conn->rest.get_ResponseStatusCode();
if (partStatus != 200) {
@@ -2392,7 +2615,7 @@ namespace ANSCENTER
const char* etag = conn->rest.responseHdrByName("ETag");
if (!conn->rest.get_LastMethodSuccess() || etag == nullptr || etag[0] == '\0') {
lastError = "Part " + std::to_string(partNumber) + ": ETag missing from response";
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return AttemptResult::Transient;
}
@@ -2410,9 +2633,8 @@ namespace ANSCENTER
responseXml = conn->rest.fullRequestString("POST", objectPath.c_str(), partsListXml.getXml());
if (!conn->rest.get_LastMethodSuccess()) {
lastError = std::string("Complete failed: ") + conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
{ AttemptResult r = ClassifyFailedRequest(std::move(conn), lastError);
lastError = "Complete: " + lastError; return r; }
}
int completeStatus = conn->rest.get_ResponseStatusCode();
if (completeStatus != 200) {
@@ -2444,17 +2666,13 @@ namespace ANSCENTER
const std::string kOp = "ANSAWSS3::UploadJpegImage";
// _logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation checks (permanent — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || bufferLength <= 0) {
_logger.LogError(kOp,
"Bucket name or buffer length is invalid",
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2486,9 +2704,7 @@ namespace ANSCENTER
CkStringBuilder sbResponse;
if (!conn->rest.FullRequestBd("PUT", objectPath.c_str(), binData, sbResponse)) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -2516,17 +2732,13 @@ namespace ANSCENTER
const std::string kOp = "ANSAWSS3::UploadPrefixJpegImage";
//_logger.LogError(kOp,"Start Uploading...",__FILE__, __LINE__);
// Early validation checks (permanent failures — do NOT retry)
if (!_isLicenseValid || !_isUnlockCodeValid || !_bConnected) {
_logger.LogError(kOp,
!_isLicenseValid ? "Invalid license" : !_isUnlockCodeValid ? "Invalid unlock code" : _retryInProgress.load() ? "Not connected (waiting for internet, retrying in background)" : "Not connected (no internet or connection not established)",
__FILE__, __LINE__);
return false;
}
if (!CheckUploadPreconditions(kOp)) return false;
if (bucketName.empty() || bufferLength <= 0 || fileName.empty()) {
_logger.LogError(kOp,
"Bucket name, buffer length, or filename is invalid",
__FILE__, __LINE__);
SetCallStatus(kCallPermanentError); // caller bug — retrying cannot help
return false;
}
@@ -2574,9 +2786,7 @@ namespace ANSCENTER
CkStringBuilder sbResponse;
if (!conn->rest.FullRequestBd("PUT", objectPath.c_str(), binData, sbResponse)) {
lastError = conn->rest.lastErrorText();
ReleaseConnection(std::move(conn));
return AttemptResult::Transient;
return ClassifyFailedRequest(std::move(conn), lastError);
}
int statusCode = conn->rest.get_ResponseStatusCode();
@@ -2764,7 +2974,7 @@ namespace ANSCENTER
if (!conn->rest.SendReqNoBody("GET", objectPath.c_str())) {
_logger.LogError("ANSAWSS3::DownloadFile - Request failed",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return false;
}
@@ -2773,7 +2983,7 @@ namespace ANSCENTER
if (responseStatusCode < 0) {
_logger.LogError("ANSAWSS3::DownloadFile - Failed to read response header",
conn->rest.lastErrorText(), __FILE__, __LINE__);
ReleaseConnection(std::move(conn));
DiscardConnection(std::move(conn)); // transport failed — do not pool a suspect socket
return false;
}