10 Commits
Author SHA1 Message Date
nghianguyenandClaude Opus 5.5 dc699f5a79 fix(EPLoader): remove the staging folders that dead processes left in %TEMP%
Each process copies the ONNX Runtime and provider DLLs (~200 MB with CUDA
and TensorRT) into %TEMP%\anscenter_ort_<pid>, and only a clean Shutdown()
deletes it. A process that is killed or crashes leaves it for good. On a
customer's training PC (2026-10-14) ANSGateway's watchdog restarted
ANSVIS_Server every minute while its REST API was down, leaving a folder a
minute in C:\Windows\Temp until the drive filled and training failed.

Before staging, each process now deletes every anscenter_ort_<pid> folder
whose PID is not a running process. A live process's folder is never
touched, since its provider DLLs load lazily from there. Both copies of
EPLoader (core/ANSLibsLoader and engines/ONNXEngine) get the sweep.

Compile-checked with MinGW against the Windows headers; not yet run on
Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 13:15:50 +11:00
nghianguyenandClaude Opus 5 cb964907f0 feat(ANSLS): know ANSMIND as 1009, and stop filing unknown products under it
The gateway's product table already runs to 1009 ANSMIND and 1010 ANSGW.
The licensing library stopped at 1008, so a real ANSMIND licence could not
be named, deactivated or listed here, while every product name it did not
recognise quietly became product 1009. With 1009 now a real product, that
fallback would mint an ANSMIND trial for a typo.

ANSPRODUCTS gains ANSMIND = 1009 and ANSGW = 1010, and both are added
wherever the product table is spelled out: ActivateTrialLicense,
DeactivateLicense, ActivateLicense, ActivateLicenseWithCustomHWID and
ListLicenses, which now reports an ANSMIND and an ANSGW row like the
gateway does.

Unknown names no longer map to any id. ActivateTrialLicense returns 0, its
existing failure value, instead of generating a key. ListLicenses skips
the row. ActivateLicense and ActivateLicenseWithCustomHWID keep passing
the name through as the registration name: they never assigned an id, the
key's own validation data rejects a name it was not issued for, and the
gateway resolves an unknown product the same way.

The ANSVIS master-licence rule in LicenseVerification is untouched.

Separately, the fallback loops in ANSUtilities, ANSAWSS3 and ANSLLM asked
for product 1008 under the registration name "ANSSRT". The product is
ANSTS, so a genuine ANSTS licence could never satisfy them. Only that
string changes; the loops still stop at 1008.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 10:38:31 +10:00
nghianguyenandClaude Opus 5 cd99fcf6d9 feat(ANSODEngine): run a sealed model on TensorRT without unsealing it
A .ansmodel reaching the GPU path went to ModelOptimizer, which knows one
way to open a model — unzip it — and failed on the first byte. Even past
that, the TensorRT engine only ever parsed an ONNX file, so the one thing
the format exists to prevent, a decrypted model on disk, was the only way
it could have run.

The builder now takes its bytes from memory when it has them. The path it
is given still names the plan that gets cached beside it, because a plan
is a built artefact and carries no model: the second run loads it and
never needs the ONNX at all. For a package the plan lands in the folder
the zip flow used, so a model converted from a zip reuses the plan that
zip already built.

OptimizeModelStr recognises a package and skips the unzip-and-optimise
leg entirely: the handle below it opens the package in memory and builds
the plan from the decrypted bytes, which is the same work by a shorter
road.

Still on the file path: the other TensorRT handlers (segmentation, pose,
classification). Only the YOLO detector reads a package today.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 09:38:45 +10:00
nghianguyenandClaude Opus 5 1e369fdfef test(ANSODEngine): let the harness run a model this machine actually has
Every test in that file names a model and a video from whichever
workstation it was written on, so anywhere else the harness prints
"Model zip file is not exist" and exits before it reaches the GPU. That
is what happened on the machine that is meant to be testing CUDA and
TensorRT.

Given a model and a video on the command line it runs those instead,
with an optional engine type and detection type. With no arguments it
behaves exactly as before, so nobody's workflow changes.

The model may be a .zip or a sealed .ansmodel, because the engine now
opens either — and that is the only way to watch a sealed model run on
a GPU, which is the last thing in this format that has never been seen
working end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 08:28:50 +10:00
nghianguyenandClaude Opus 5 b69a0fbcba fix(build): find nlohmann/json where the machine actually keeps it
The package library's Windows branch pointed at vcpkg alone. On a real
Windows build machine the header lives in the ANSLibs tree, so anything
linking this library failed with "Cannot open include file:
'nlohmann/json.hpp'" while the engines compiled fine — they add ANSLibs
to their own include path and never noticed.

Now it is searched for by name, with the ANSLibs tree, vcpkg and the
usual system paths as hints, and the configure says where it found it.
One code path for every platform instead of an assumption per platform.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 08:00:51 +10:00
nghianguyenandClaude Opus 5 e76d3b81e2 fix(cuda): spell the fast-math flag the way nvcc spells it
Every .cu file failed on CUDA 13 with "nvcc fatal: Unknown option
'--use-fast-math'". The flag is --use_fast_math, with underscores; older
toolkits accepted the hyphenated form as well and this one does not, so
the engine layer could not compile a single kernel on a current CUDA.

Found on the Windows machine with CUDA 13.0, where it stopped the build
at the first two kernels.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 07:58:49 +10:00
nghianguyenandClaude Opus 5 eb84d48e5c fix(build): do not demand a policy older CMake has never heard of
Line two set CMP0167 unconditionally. That policy arrived in CMake 3.30
and this project asks for 3.28, so every CMake between the two failed
the configure at line two with "Policy CMP0167 is not known to this
version" — before it had done anything at all. Found on the Windows
machine that builds the CUDA engines, where it stopped the build dead.

Guarded, so a newer CMake still gets the quiet FindBoost the policy was
set for and an older one configures instead of refusing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 07:46:43 +10:00
nghianguyenandClaude Opus 5 3798e3513b fix(YOLO12OD): decode an end-to-end head instead of reading it as a grid
This engine took the feature count from outputShape[1] unconditionally.
For a yolo26 export with end2end on — [1, 300, 6], rows of
(x1 y1 x2 y2 score class) with the suppression already inside the graph
— that means 296 classes, coordinates read as class scores, and boxes
covering the frame. No error, just nonsense, which is exactly what the
Labelling Tool produced from the same model before its decoder was
fixed.

Its sibling ANSONNXYOLO has recognised that head from the tensor shape
for a while; this one had not, and the backend marker cannot tell them
apart because it names the runtime rather than the head.

The branch fires only where the tensor cannot be a grid: the 6 must be
the last axis with a detection count before it, hundreds rather than the
thousands of anchors a two-class [1, 6, 8400] export carries.

The decoder reuses this engine's own scaleCoords rather than repeating
the letterbox arithmetic, so both heads undo the padding the same way
and agree about where a box is, and the rounding into the integral box
type is spelled out instead of left to a narrowing conversion.

NOT COMPILED. The engine layer needs CUDA and the ANSLibs tree, neither
of which exists on this machine, so this is verified by reading: the
shape rule matches the one the other three products now share, every
symbol it uses is declared in this translation unit, and the arithmetic
is delegated to the function the grid path already trusts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 06:55:45 +10:00
nghianguyenandClaude Opus 5 ea877a61bc fix(keys): decide at configure time whether this build has the key
Whether the model-package key is present was an __has_include test
inside the header, and that test leaves no dependency for the build
system. A key emitted after a build had already compiled the file would
not take effect until someone cleaned, and the resulting engine looks
exactly like one built without a key — which on a release machine means
shipping an engine that silently cannot open a package.

CMake now decides it and says which case it is, so emitting the key and
re-configuring is what installs it, visibly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 05:56:58 +10:00
nghianguyenandClaude Opus 5 3c9e0b79f5 fix(ANSModelPackage): read the head off the graph, not off the package's word
A package whose family says "yolo" may hold an end-to-end export. Boxes
took the family from the backend marker, and the marker names the
runtime, not the head — so a fast yolo26 export arrived declared as
plain yolo while its graph emits [1, N, 6] with the NMS inside it.
Decoding that as a [1, 4+nc, N] grid reads coordinates as confidences:
detections everywhere, and nothing that looks like a failure.

OpenModelPackage now reads the first output's dims from the model it has
just decrypted, and corrects the family with the reason in the warning
rather than swapping it silently. It costs nothing there: the bytes are
in hand. The legacy marker is left alone, so anything converting back to
a zip still writes what ANSCORE expects.

A two-class v8 export is [1, 6, 8400] and must not be mistaken for one,
which is why the 6 has to be the last axis with a detection count before
it. Both cases are tested, and the package this Mac exported before the
writer was fixed now reads as yolo10 here too. 46 checks, no failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-13 00:19:15 +10:00
23 changed files with 766 additions and 137 deletions
+13 -3
View File
@@ -1,5 +1,11 @@
cmake_minimum_required(VERSION 3.28)
cmake_policy(SET CMP0167 NEW) # Suppress FindBoost removal warning
# CMP0167 (FindBoost removed from CMake) only exists from 3.30, and this
# project asks for 3.28 — so setting it unconditionally makes every older
# CMake fail at line two with "policy is not known to this version". Guarded,
# a newer CMake still gets the quiet FindBoost it was added for.
if(POLICY CMP0167)
cmake_policy(SET CMP0167 NEW)
endif()
# ── CUDA is optional (not available on macOS) ───────────────────
include(CheckLanguage)
@@ -17,10 +23,14 @@ if(CMAKE_CUDA_COMPILER)
set(CMAKE_CUDA_ARCHITECTURES
75-real 80-real 86-real 87-real 88-real 89-real 90-real 100-real 120-real
75-virtual)
# --use_fast_math, with underscores: that is nvcc's own spelling. Older
# toolkits also accepted the hyphenated form; CUDA 13 does not, and every
# .cu file fails with "Unknown option '--use-fast-math'" before it compiles
# a line.
if(MSVC)
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use-fast-math -Xcompiler=/utf-8")
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math -Xcompiler=/utf-8")
else()
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use-fast-math")
set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --use_fast_math")
endif()
else()
project(ANSCORE LANGUAGES C CXX)
+60
View File
@@ -325,6 +325,65 @@ namespace ANSCENTER {
RemoveDirectoryA(dir.c_str());
std::cout << "[EPLoader] Temp staging dir deleted: " << dir << std::endl;
}
// ── SweepStaleTempDirs ────────────────────────────────────────────────────
// Removes the staging folders that dead processes left behind.
//
// Only a clean Shutdown() deletes a process's folder. A process that is
// killed or crashes leaves its ~200 MB of DLLs in %TEMP% for good, and a
// service restarted in a loop (ANSGateway's ANSVIS watchdog restarts
// ANSVIS_Server every minute while its REST API is down) fills the system
// drive: C:\Windows\Temp grew by a folder a minute on a customer's
// training PC until training failed for lack of disk.
//
// A folder is removed only when its PID is not a running process. A live
// process's folder is never touched, even partly: its provider DLLs
// (TensorRT, CUDA) load lazily from there, after start-up. A PID reused by
// an unrelated process keeps its stale folder until that one exits —
// rare, and bounded.
static bool ProcessIsRunning(DWORD pid)
{
HANDLE h = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
if (!h) {
// No such process. Anything else (access denied: another user's
// process, a protected one) means it exists.
return GetLastError() != ERROR_INVALID_PARAMETER;
}
DWORD code = 0;
const bool running = GetExitCodeProcess(h, &code) && code == STILL_ACTIVE;
CloseHandle(h);
return running;
}
static void SweepStaleTempDirs()
{
char tmp[MAX_PATH] = {};
if (!GetTempPathA(MAX_PATH, tmp)) return;
const std::string base(tmp);
const std::string prefix = "anscenter_ort_";
const DWORD self = GetCurrentProcessId();
WIN32_FIND_DATAA fd{};
HANDLE hFind = FindFirstFileA((base + prefix + "*").c_str(), &fd);
if (hFind == INVALID_HANDLE_VALUE) return;
std::vector<std::string> stale;
do {
if (!(fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)) continue;
const std::string name(fd.cFileName);
const std::string digits = name.substr(prefix.size());
if (digits.empty() || digits.size() > 10 ||
digits.find_first_not_of("0123456789") != std::string::npos)
continue;
const unsigned long long pid = std::stoull(digits);
if (pid == 0 || pid > 0xFFFFFFFFull || DWORD(pid) == self) continue;
if (!ProcessIsRunning(DWORD(pid))) stale.push_back(base + name);
} while (FindNextFileA(hFind, &fd));
FindClose(hFind);
for (const std::string& dir : stale) DeleteTempDir(dir);
if (!stale.empty())
std::cout << "[EPLoader] Removed " << stale.size()
<< " staging dir(s) left by processes that are gone." << std::endl;
}
#endif // _WIN32
@@ -345,6 +404,7 @@ namespace ANSCENTER {
#ifdef _WIN32
// ── Windows: stage ALL ep_dir DLLs into a process-unique temp folder ────
SweepStaleTempDirs();
std::string temp_dir = MakeTempDir();
std::cout << "[EPLoader] Staging dir : " << temp_dir << std::endl;
+47 -4
View File
@@ -1777,10 +1777,19 @@ namespace ANSCENTER
registrationName = "ANSTS";
productId = 1008;
}
else {
registrationName = productName;
else if (productName == "ANSMIND") {//1009
registrationName = "ANSMIND";
productId = 1009;
}
else if (productName == "ANSGW") {//1010
registrationName = "ANSGW";
productId = 1010;
}
else {
// Unknown product: refuse rather than mint a trial under some real product's id.
std::cout << "ActivateTrialLicense: unknown product name '" << productName << "'" << std::endl;
return 0;
}
std::string trialLicenseKey = GetTrialLicenseKey(registrationName);
if (trialLicenseKey.empty()) {
@@ -1849,6 +1858,12 @@ namespace ANSCENTER
case ANSCENTER::ANSPRODUCTS::ANS_TS://1008
_productName = "ANSTS";
break;
case ANSCENTER::ANSPRODUCTS::ANSMIND://1009
_productName = "ANSMIND";
break;
case ANSCENTER::ANSPRODUCTS::ANSGW://1010
_productName = "ANSGW";
break;
}
if (_productName == productName) {
// Delete the license file
@@ -1896,6 +1911,12 @@ namespace ANSCENTER
else if (productName == "ANSTS") {//1008
registrationName = "ANSTS";
}
else if (productName == "ANSMIND") {//1009
registrationName = "ANSMIND";
}
else if (productName == "ANSGW") {//1010
registrationName = "ANSGW";
}
else {
registrationName = productName;
}
@@ -1936,6 +1957,12 @@ namespace ANSCENTER
else if (productName == "ANSTS") {//1008
registrationName = "ANSTS";
}
else if (productName == "ANSMIND") {//1009
registrationName = "ANSMIND";
}
else if (productName == "ANSGW") {//1010
registrationName = "ANSGW";
}
else {
registrationName = productName;
}
@@ -2002,6 +2029,8 @@ namespace ANSCENTER
ansLicenseCollection.push_back("ANSALPR");//1006
ansLicenseCollection.push_back("ANSCV");//1007
ansLicenseCollection.push_back("ANSTS");//1008
ansLicenseCollection.push_back("ANSMIND");//1009
ansLicenseCollection.push_back("ANSGW");//1010
for (auto& element : ansLicenseCollection) {
boost::property_tree::ptree dataNode;
@@ -2064,6 +2093,12 @@ namespace ANSCENTER
case ANSCENTER::ANSPRODUCTS::ANS_TS:
_productName = "ANSTS";
break;
case ANSCENTER::ANSPRODUCTS::ANSMIND:
_productName = "ANSMIND";
break;
case ANSCENTER::ANSPRODUCTS::ANSGW:
_productName = "ANSGW";
break;
}
if (_productName == element) {
_registrationName = ANSUtility::DecodeBase64Utility(_validationData.c_str());// Decode
@@ -2141,10 +2176,18 @@ namespace ANSCENTER
registrationName = "ANSTS";
productId = 1008;
}
else {
registrationName = ansLicenseInfo.productName;
else if (ansLicenseInfo.productName == "ANSMIND") {
registrationName = "ANSMIND";
productId = 1009;
}
else if (ansLicenseInfo.productName == "ANSGW") {
registrationName = "ANSGW";
productId = 1010;
}
else {
// Unknown product: skip the row rather than report it under some real product's id.
continue;
}
dataNode.put("product_name", ansLicenseInfo.productName);
std::string trialLicenseKey = GetTrialLicenseKey(registrationName);
if (trialLicenseKey.empty()) {
+3 -1
View File
@@ -259,7 +259,9 @@ namespace ANSCENTER
ANS_OCR = 1005,
ANS_ALPR =1006,
ANS_CV = 1007,
ANS_TS = 1008
ANS_TS = 1008,
ANSMIND = 1009,
ANSGW = 1010
};
enum ANSLICENSESTATUS {
ACTIVE = 0,
+7 -4
View File
@@ -13,11 +13,14 @@
// package is refused with a sentence that says why. That is what keeps the key
// out of a developer's checkout without keeping developers out of the code.
#if defined(__has_include)
#if __has_include("ANSModelKeysGenerated.h")
// ANSCORE_HAVE_BUILTIN_MODEL_KEYS is set by CMake when the ceremony's
// generated header is there. It is not tested with __has_include on purpose:
// that test leaves no dependency for the build system, so a key emitted after
// a build would not take effect until someone cleaned — and the failure looks
// exactly like a missing key, which on a release machine means shipping an
// engine that silently cannot open a package.
#if defined(ANSCORE_HAVE_BUILTIN_MODEL_KEYS)
#include "ANSModelKeysGenerated.h"
#define ANSCORE_HAVE_BUILTIN_MODEL_KEYS 1
#endif
#endif
#include "ANSModelPackage.h"
+21 -9
View File
@@ -217,16 +217,28 @@ inline bool OpenModelPackage(const std::string& path, OpenedModel& out,
out.name = m.value("project", std::string());
if (out.name.empty()) out.name = path;
// A package written before manifests carried an input size still has to
// answer, and the model in hand is the thing that knows. It costs nothing
// here: the bytes are already decrypted.
if (out.inputWidth <= 0 || out.inputHeight <= 0) {
onnxshape::InputShape shape;
// WHAT AN OLDER PACKAGE CANNOT TELL US, and its model can. The input
// size, when the manifest predates the field; and the head, because a
// package whose family is "yolo" may hold an end-to-end export — the box
// used to take the family from the backend marker, and the marker names
// the runtime rather than the head. It costs nothing to ask here: the
// bytes are already decrypted.
{
onnxshape::GraphShape g;
std::string swhy;
if (onnxshape::fromMemory(out.model.data(), out.model.size(), shape,
swhy)) {
out.inputWidth = shape.width;
out.inputHeight = shape.height;
if (onnxshape::graphFromMemory(out.model.data(), out.model.size(), g,
swhy)) {
if (out.inputWidth <= 0 || out.inputHeight <= 0) {
out.inputWidth = g.input.width;
out.inputHeight = g.input.height;
}
if (g.endToEnd() && out.family == "yolo") {
out.family = "yolo10";
out.warning = out.warning.empty()
? "this package says family yolo, but its head is "
"end-to-end; reading it as yolo10"
: out.warning;
}
}
}
return true;
+145 -32
View File
@@ -3,18 +3,16 @@
#pragma once
// ANSOnnxShape — the input geometry a model declares, read off the graph.
//
// WHY A READER NEEDS THIS. An engine has to know what its model expects before
// it can preprocess a single frame, and that answer has always come from
// whatever the bundle happened to carry: model_config.json's input_size, or an
// `input` object in a package manifest. Neither is present on every model —
// most of the bundles in the field are classes.names, a marker file and the
// export, and nothing else. The graph is always there, and the graph is what
// the session is built from, so where the metadata is silent the model itself
// is asked instead of every engine falling back to 640x640.
// WHY A READER NEEDS THIS. A node has to know what its model expects before it
// can preprocess a single frame, and until now that answer came from whatever
// the bundle happened to carry: model_config.json's input_size in a folder, an
// `input` object in a package manifest. Neither is present on every model. The
// graph always is, and the graph is the thing the session is built from — so
// where the metadata is silent, ask the model itself rather than assume 640.
//
// TWO SOURCES, ONE PARSER. A package's model is already in memory, decrypted,
// and must never reach the disk; an inflated bundle's is a file that may be
// hundreds of megabytes. The template below reads either, and the file case
// and must never reach the disk; an imported folder's model is a file that may
// be hundreds of megabytes. The template below reads either, and the file case
// SEEKS past the nodes and the initializers rather than reading them — the
// fields this wants sit after all of the weight on the wire.
//
@@ -23,12 +21,16 @@
// ModelProto.graph field 7, length-delimited
// GraphProto.input field 11, repeated (after node and
// initializer on the wire)
// GraphProto.output field 12, repeated
// ValueInfoProto.type field 2
// TypeProto.tensor_type field 1
// TypeProto.Tensor.shape field 2
// TensorShapeProto.dim field 1, repeated
// Dimension.dim_value field 1, varint (dim_param means the
// axis is dynamic: reported as 0)
// ModelProto.metadata_props field 14, repeated StringStringEntryProto
// StringStringEntryProto.key field 1, string
// StringStringEntryProto.value field 2, string
//
// An axis the graph leaves open is reported as zero, and a file that cannot be
// parsed is reported as a failure. Both mean the same thing to a caller: this
@@ -36,6 +38,7 @@
// model's input size is worse than being silent about it.
#include <cstdint>
#include <cctype>
#include <cstdio>
#include <filesystem>
#include <fstream>
@@ -52,6 +55,32 @@ struct InputShape {
bool known() const { return height > 0 && width > 0; }
};
// What the whole graph says about itself: what goes in, and what shape comes
// out — which is the difference between a head that needs decoding and one
// that has already done it.
struct GraphShape {
InputShape input;
std::vector<int64_t> output; // the FIRST output's dims; empty when unknown
bool exportSaysEndToEnd = false; // the model's own metadata, when it says
// AN END-TO-END HEAD, [1, N, 6]: rows of (x1 y1 x2 y2 score class) with the
// NMS already inside the graph. Telling one apart from an ordinary YOLO head
// matters more than it looks: feeding [1,N,6] to a v8 decoder yields boxes
// everywhere with coordinates read as confidences — plausible-looking
// garbage rather than an error.
//
// The trap is a two-class v8 export, whose [1, 4+nc, N] is [1, 6, 8400] —
// a 6 in the shape, and not end-to-end. So the 6 must be the LAST axis, and
// the axis before it must be a detection count rather than an anchor count;
// an end-to-end head emits max_det rows, which is hundreds, never thousands.
// When the export's own metadata says end2end, that settles it outright.
bool endToEnd() const {
if (exportSaysEndToEnd) return true;
return output.size() == 3 && output[0] == 1 && output[2] == 6 &&
output[1] > 0 && output[1] <= 1000;
}
};
namespace detail {
// A model already in memory: the decrypted contents of a package.
@@ -158,51 +187,105 @@ std::vector<int64_t> dims(Src& s, int64_t end) {
return out;
}
// One ValueInfoProto (a graph input or output), given a stream on its payload:
// its dims, or empty when it is not a shaped tensor.
template <class Src>
bool read(Src& s, int64_t end, InputShape& out, std::string& why) {
std::vector<int64_t> valueDims(Src& s, int64_t valueEnd) {
uint64_t len = 0;
if (!seekField(s, valueEnd, 2, len)) return {}; // ValueInfoProto.type
const int64_t typeEnd = s.tell() + int64_t(len);
if (!seekField(s, typeEnd, 1, len)) return {}; // TypeProto.tensor_type
const int64_t tensorEnd = s.tell() + int64_t(len);
if (!seekField(s, tensorEnd, 2, len)) return {}; // Tensor.shape
return dims(s, s.tell() + int64_t(len));
}
// The graph's first input and first output, and whether the export declared
// itself end-to-end. `end` is the whole file; `graphStart` lets the output be
// found after the input, since both hang off the same graph.
template <class Src>
bool readGraph(Src& s, int64_t end, GraphShape& out, std::string& why) {
uint64_t len = 0;
if (!seekField(s, end, 7, len)) { // ModelProto.graph
why = "this does not look like an ONNX model";
return false;
}
const int64_t graphEnd = s.tell() + int64_t(len);
const int64_t graphStart = s.tell();
const int64_t graphEnd = graphStart + int64_t(len);
if (!seekField(s, graphEnd, 11, len)) { // GraphProto.input
why = "the graph declares no input";
return false;
}
const int64_t inputEnd = s.tell() + int64_t(len);
if (!seekField(s, inputEnd, 2, len)) { // ValueInfoProto.type
why = "the input declares no type";
return false;
const std::vector<int64_t> in = valueDims(s, s.tell() + int64_t(len));
// The output, from the graph's start again: it sits after the input on the
// wire, but seeking from the start costs nothing and does not depend on it.
s.seek(graphStart);
if (seekField(s, graphEnd, 12, len)) // GraphProto.output
out.output = valueDims(s, s.tell() + int64_t(len));
// The export's own word on its head, which beats any shape rule.
s.seek(0);
while (s.ok() && s.tell() >= 0 && s.tell() < end) {
uint32_t field = 0, wire = 0;
if (!tag(s, field, wire)) break;
if (field != 14 || wire != 2) { // ModelProto.metadata_props
if (!skip(s, wire)) break;
continue;
}
uint64_t entryLen = 0;
if (!varint(s, entryLen)) break;
const int64_t entryEnd = s.tell() + int64_t(entryLen);
std::string key, value;
while (s.ok() && s.tell() >= 0 && s.tell() < entryEnd) {
uint32_t ef = 0, ew = 0;
if (!tag(s, ef, ew)) break;
if ((ef == 1 || ef == 2) && ew == 2) {
uint64_t n = 0;
if (!varint(s, n)) break;
std::string text;
for (uint64_t i = 0; i < n; ++i) {
const int c = s.get();
if (c < 0) break;
text.push_back(char(c));
}
(ef == 1 ? key : value) = text;
} else if (!skip(s, ew)) {
break;
}
}
s.seek(entryEnd);
if (key == "end2end") {
for (char& c : value) c = char(::tolower(static_cast<unsigned char>(c)));
out.exportSaysEndToEnd = (value == "true" || value == "1");
}
}
const int64_t typeEnd = s.tell() + int64_t(len);
if (!seekField(s, typeEnd, 1, len)) { // TypeProto.tensor_type
why = "the input is not a tensor";
return false;
}
const int64_t tensorEnd = s.tell() + int64_t(len);
if (!seekField(s, tensorEnd, 2, len)) { // Tensor.shape
why = "the input tensor has no shape";
return false;
}
const std::vector<int64_t> d = dims(s, s.tell() + int64_t(len));
// NCHW is the only layout these engines run.
if (d.size() != 4) {
if (in.size() != 4) {
why = "the input is not a four-dimensional image tensor";
return false;
}
auto axis = [](int64_t v) { return (v > 0 && v < 1000000) ? int(v) : 0; };
out.channels = axis(d[1]);
out.height = axis(d[2]);
out.width = axis(d[3]);
if (!out.known()) {
out.input.channels = axis(in[1]);
out.input.height = axis(in[2]);
out.input.width = axis(in[3]);
if (!out.input.known()) {
why = "the model takes a dynamic input size";
return false;
}
return true;
}
template <class Src>
bool read(Src& s, int64_t end, InputShape& out, std::string& why) {
GraphShape g;
if (!readGraph(s, end, g, why)) return false;
out = g.input;
return true;
}
} // namespace detail
// What a model already in memory declares.
@@ -216,6 +299,17 @@ inline bool fromMemory(const void* data, size_t size, InputShape& out,
return detail::read(s, int64_t(size), out, why);
}
// The whole graph, from a model already in memory.
inline bool graphFromMemory(const void* data, size_t size, GraphShape& out,
std::string& why) {
if (!data || size == 0) {
why = "there is no model to read";
return false;
}
detail::MemSource s{static_cast<const uint8_t*>(data), int64_t(size), 0};
return detail::readGraph(s, int64_t(size), out, why);
}
// What a model on disk declares. The file is walked, not loaded.
inline bool fromFile(const std::string& path, InputShape& out,
std::string& why) {
@@ -235,6 +329,25 @@ inline bool fromFile(const std::string& path, InputShape& out,
return detail::read(s, int64_t(size), out, why);
}
// The whole graph, from a model on disk. Walked, not loaded.
inline bool graphFromFile(const std::string& path, GraphShape& out,
std::string& why) {
std::error_code ec;
const auto size = std::filesystem::file_size(path, ec);
if (ec || size == 0) {
why = "the model is not there to read";
return false;
}
detail::FileSource s;
s.f.open(path, std::ios::binary);
s.size = int64_t(size);
if (!s.f) {
why = "the model could not be opened";
return false;
}
return detail::readGraph(s, int64_t(size), out, why);
}
} // namespace onnxshape
} // namespace ANSCENTER
+23 -11
View File
@@ -14,18 +14,30 @@ target_include_directories(ANSModelPackage PUBLIC
${CMAKE_SOURCE_DIR}/3rdparty/monocypher
)
# nlohmann/json comes from vcpkg on Windows and the system elsewhere, the same
# way ANSFR already takes it.
if(WIN32)
target_include_directories(ANSModelPackage PUBLIC ${VCPKG_INCLUDE_DIR})
# nlohmann/json, wherever this machine keeps it: the ANSLibs tree on Windows,
# vcpkg, or the system elsewhere. Asked for by name rather than assumed — the
# Windows branch used to point at vcpkg alone, and on a real build machine the
# header lives under ANSLibs, so every target that linked this one failed to
# find it while the engines, which add ANSLibs themselves, compiled fine.
find_path(ANSCORE_NLOHMANN_INCLUDE nlohmann/json.hpp
HINTS ${ANSLIBS_DIR} ${VCPKG_INCLUDE_DIR}
/opt/homebrew/include /usr/local/include /usr/include)
if(ANSCORE_NLOHMANN_INCLUDE)
target_include_directories(ANSModelPackage PUBLIC ${ANSCORE_NLOHMANN_INCLUDE})
message(STATUS "ANSModelPackage: nlohmann/json at ${ANSCORE_NLOHMANN_INCLUDE}")
else()
find_path(ANSCORE_NLOHMANN_INCLUDE nlohmann/json.hpp
HINTS /opt/homebrew/include /usr/local/include /usr/include)
if(ANSCORE_NLOHMANN_INCLUDE)
target_include_directories(ANSModelPackage PUBLIC ${ANSCORE_NLOHMANN_INCLUDE})
else()
message(WARNING "nlohmann/json.hpp not found — ANSModelPackage will not build")
endif()
message(WARNING "nlohmann/json.hpp not found — ANSModelPackage will not build")
endif()
# The model-package key, decided at configure time: see ANSModelKeys.h for why
# this is not an __has_include test. PUBLIC so every engine that opens a
# package is compiled with the same answer.
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/ANSModelKeysGenerated.h")
target_compile_definitions(ANSModelPackage PUBLIC ANSCORE_HAVE_BUILTIN_MODEL_KEYS=1)
message(STATUS "ANSModelPackage: key present — packages can be opened")
else()
message(STATUS "ANSModelPackage: no key — packages cannot be opened "
"(run ansmodel-keys emit --product anscore, then re-run cmake)")
endif()
set_target_properties(ANSModelPackage PROPERTIES POSITION_INDEPENDENT_CODE ON)
+60
View File
@@ -383,6 +383,65 @@ namespace ANSCENTER {
RemoveDirectoryA(dir.c_str());
std::cout << "[EPLoader] Temp staging dir deleted: " << dir << std::endl;
}
// ── SweepStaleTempDirs ────────────────────────────────────────────────────
// Removes the staging folders that dead processes left behind.
//
// Only a clean Shutdown() deletes a process's folder. A process that is
// killed or crashes leaves its ~200 MB of DLLs in %TEMP% for good, and a
// service restarted in a loop (ANSGateway's ANSVIS watchdog restarts
// ANSVIS_Server every minute while its REST API is down) fills the system
// drive: C:\Windows\Temp grew by a folder a minute on a customer's
// training PC until training failed for lack of disk.
//
// A folder is removed only when its PID is not a running process. A live
// process's folder is never touched, even partly: its provider DLLs
// (TensorRT, CUDA) load lazily from there, after start-up. A PID reused by
// an unrelated process keeps its stale folder until that one exits —
// rare, and bounded.
static bool ProcessIsRunning(DWORD pid)
{
HANDLE h = OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, FALSE, pid);
if (!h) {
// No such process. Anything else (access denied: another user's
// process, a protected one) means it exists.
return GetLastError() != ERROR_INVALID_PARAMETER;
}
DWORD code = 0;
const bool running = GetExitCodeProcess(h, &code) && code == STILL_ACTIVE;
CloseHandle(h);
return running;
}
static void SweepStaleTempDirs()
{
char tmp[MAX_PATH] = {};
if (!GetTempPathA(MAX_PATH, tmp)) return;
const std::string base(tmp);
const std::string prefix = "anscenter_ort_";
const DWORD self = GetCurrentProcessId();
WIN32_FIND_DATAA fd{};
HANDLE hFind = FindFirstFileA((base + prefix + "*").c_str(), &fd);
if (hFind == INVALID_HANDLE_VALUE) return;
std::vector<std::string> stale;
do {
if (!(fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)) continue;
const std::string name(fd.cFileName);
const std::string digits = name.substr(prefix.size());
if (digits.empty() || digits.size() > 10 ||
digits.find_first_not_of("0123456789") != std::string::npos)
continue;
const unsigned long long pid = std::stoull(digits);
if (pid == 0 || pid > 0xFFFFFFFFull || DWORD(pid) == self) continue;
if (!ProcessIsRunning(DWORD(pid))) stale.push_back(base + name);
} while (FindNextFileA(hFind, &fd));
FindClose(hFind);
for (const std::string& dir : stale) DeleteTempDir(dir);
if (!stale.empty())
std::cout << "[EPLoader] Removed " << stale.size()
<< " staging dir(s) left by processes that are gone." << std::endl;
}
#endif // _WIN32
@@ -418,6 +477,7 @@ namespace ANSCENTER {
// so it resolves providers relative to %TEMP% — where they don't exist.
// Staging ALL provider DLLs alongside our renamed copy fixes this.
SweepStaleTempDirs();
std::string temp_dir = MakeTempDir();
std::cout << "[EPLoader] Staging dir : " << temp_dir << std::endl;
@@ -316,7 +316,7 @@ bool Engine<T>::buildLoadNetwork(std::string onnxModelPath, const std::array<flo
return false;
}
}
if (!FileExist(onnxModelPath)) {
if (!hasOnnxBytes() && !FileExist(onnxModelPath)) {
// ONNX model does not exist, try to find alternative precision engine
std::cout << "Searching for alternative precision engine..." << std::endl;
@@ -390,12 +390,9 @@ bool Engine<T>::buildLoadNetwork(std::string onnxModelPath, const std::array<flo
auto tempNetwork = std::unique_ptr<nvinfer1::INetworkDefinition>(TRT_CREATE_NETWORK(tempBuilder));
auto tempParser = std::unique_ptr<nvonnxparser::IParser>(nvonnxparser::createParser(*tempNetwork, m_logger));
std::ifstream onnxFile(onnxModelPath, std::ios::binary | std::ios::ate);
std::streamsize onnxSize = onnxFile.tellg();
onnxFile.seekg(0, std::ios::beg);
std::vector<char> onnxBuffer(onnxSize);
if (!onnxFile.read(onnxBuffer.data(), onnxSize)) {
std::cout << "Error: Failed to read ONNX file" << std::endl;
std::vector<char> onnxBuffer;
if (!readOnnxBytes(onnxModelPath, onnxBuffer)) {
std::cout << "Error: Failed to read ONNX model" << std::endl;
return false;
}
@@ -1465,7 +1462,7 @@ bool Engine<T>::build(std::string onnxModelPath, const std::array<float, 3>& sub
std::cout << "Engine file already exists: " << engineName << std::endl;
return true;
}
if (!FileExist(onnxModelPath)) {
if (!hasOnnxBytes() && !FileExist(onnxModelPath)) {
std::cout << "Error: ONNX model file does not exist: " << onnxModelPath << std::endl;
return false;
}
@@ -1502,16 +1499,14 @@ bool Engine<T>::build(std::string onnxModelPath, const std::array<float, 3>& sub
return false;
}
// Read the onnx file into memory
std::ifstream file(onnxModelPath, std::ios::binary | std::ios::ate);
std::streamsize size = file.tellg();
file.seekg(0, std::ios::beg);
std::vector<char> buffer(size);
if (!file.read(buffer.data(), size)) {
std::cout << "Error: Unable to read ONNX file" << std::endl;
// Get the onnx bytes: from the in-memory buffer of a sealed package, or
// from the file on disk.
std::vector<char> buffer;
if (!readOnnxBytes(onnxModelPath, buffer)) {
std::cout << "Error: Unable to read ONNX model" << std::endl;
return false;
}
const std::streamsize size = static_cast<std::streamsize>(buffer.size());
std::cout << "ONNX model size: " << size / (1024 * 1024) << " MiB" << std::endl;
@@ -2471,16 +2466,14 @@ bool Engine<T>::buildWithRetry(std::string onnxModelPath,
auto tempParser = std::unique_ptr<nvonnxparser::IParser>(
nvonnxparser::createParser(*tempNetwork, m_logger));
std::ifstream onnxFile(onnxModelPath, std::ios::binary | std::ios::ate);
if (!onnxFile.good()) {
std::vector<char> onnxBuffer;
const bool onnxReadable = readOnnxBytes(onnxModelPath, onnxBuffer);
if (!onnxReadable) {
ANS_DBG("TRT_Build", "buildWithRetry WARN cannot open ONNX for pre-parse path=%s",
onnxModelPath.c_str());
}
if (onnxFile.good()) {
std::streamsize onnxSize = onnxFile.tellg();
onnxFile.seekg(0, std::ios::beg);
std::vector<char> onnxBuffer(onnxSize);
if (onnxFile.read(onnxBuffer.data(), onnxSize)) {
if (onnxReadable) {
{
unsigned long sehRetryParse = 0;
bool retryParsed = parseOnnxModelSafe(tempParser.get(),
onnxBuffer.data(), onnxBuffer.size(), &sehRetryParse);
@@ -2508,9 +2501,6 @@ bool Engine<T>::buildWithRetry(std::string onnxModelPath,
(int)retryParsed,
retryParsed ? tempNetwork->getNbInputs() : -1);
}
} else {
ANS_DBG("TRT_Build", "buildWithRetry WARN ONNX read failed bytes=%zd",
(long long)onnxSize);
}
}
} else {
@@ -88,8 +88,20 @@ public:
const std::array<float, 3>& subVals,
const std::array<float, 3>& divVals,
bool normalize,
int maxSlotsPerGpu)
int maxSlotsPerGpu,
const void* onnxData = nullptr,
size_t onnxSize = 0)
{
// A sealed .ansmodel is decrypted in memory and never written out, so
// the caller hands the bytes down here: every Engine this call builds
// parses them instead of opening modelPath, which only names the
// TensorRT plan that gets cached.
auto newEngine = [&]() {
auto e = std::make_shared<Engine<T>>(options);
if (onnxData != nullptr && onnxSize > 0) e->setOnnxBytes(onnxData, onnxSize);
return e;
};
// Optimizer / temporary engines: maxSlotsPerGpu==0 means the caller
// only needs a lightweight, non-shared engine (e.g., OptimizeModelStr).
// Bypass the pool cache entirely:
@@ -99,7 +111,7 @@ public:
// Note: maxSlotsPerGpu==1 is now the normal "1 slot per GPU" multi-GPU
// round-robin mode, so it goes through the pool path below.
if (maxSlotsPerGpu == 0) {
auto engine = std::make_shared<Engine<T>>(options);
auto engine = newEngine();
bool ok = engine->buildLoadNetwork(modelPath, subVals, divVals, normalize);
return ok ? engine : nullptr;
}
@@ -141,13 +153,13 @@ public:
}
// Cache miss — create new Engine pool
auto engine = std::make_shared<Engine<T>>(options);
auto engine = newEngine();
bool ok = engine->buildLoadNetwork(modelPath, subVals, divVals, normalize, maxSlotsPerGpu);
if (!ok) {
// Step 1: Force-evict all pools with refcount=0 to reclaim VRAM
int evicted = forceEvictPending();
if (evicted > 0) {
engine = std::make_shared<Engine<T>>(options);
engine = newEngine();
ok = engine->buildLoadNetwork(modelPath, subVals, divVals, normalize, maxSlotsPerGpu);
}
@@ -156,7 +168,7 @@ public:
// consumes ~300-500 MB vs ~50-100 MB for a simple loadNetwork.
// Lightweight mode: tasks queue for a single shared slot — slower but works.
if (!ok) {
engine = std::make_shared<Engine<T>>(options);
engine = newEngine();
ok = engine->buildLoadNetwork(modelPath, subVals, divVals, normalize);
}
@@ -183,7 +195,7 @@ public:
}
// Final retry — try lightweight again after delay
engine = std::make_shared<Engine<T>>(options);
engine = newEngine();
ok = engine->buildLoadNetwork(modelPath, subVals, divVals, normalize);
}
@@ -4,6 +4,48 @@
#include <NvInferVersion.h> // also defines TRT version macros
#include <cudnn_version.h> // CUDNN_MAJOR/MINOR/PATCHLEVEL
#include <cuda_runtime.h> // cudaRuntimeGetVersion
#include <cstring> // memset for wiping decrypted model bytes
#include <fstream>
// ============================================================================
// ONNX bytes in memory
//
// A sealed .ansmodel is decrypted into RAM and stays there: handing the
// parser a buffer is what keeps the plaintext model off the disk. The build
// path still takes a path, because the path names the TensorRT plan that is
// cached beside it — a plan is a built artefact and carries no model.
// ============================================================================
template <typename T>
void Engine<T>::setOnnxBytes(const void* data, size_t size) {
clearOnnxBytes();
if (data == nullptr || size == 0) return;
const char* p = static_cast<const char*>(data);
m_onnxBytes.assign(p, p + size);
}
template <typename T>
void Engine<T>::clearOnnxBytes() {
if (m_onnxBytes.empty()) return;
std::memset(m_onnxBytes.data(), 0, m_onnxBytes.size());
m_onnxBytes.clear();
m_onnxBytes.shrink_to_fit();
}
template <typename T>
bool Engine<T>::readOnnxBytes(const std::string& onnxModelPath,
std::vector<char>& buffer) const {
if (!m_onnxBytes.empty()) {
buffer = m_onnxBytes;
return true;
}
std::ifstream file(onnxModelPath, std::ios::binary | std::ios::ate);
if (!file.good()) return false;
const std::streamsize size = file.tellg();
if (size <= 0) return false;
file.seekg(0, std::ios::beg);
buffer.resize(static_cast<size_t>(size));
return static_cast<bool>(file.read(buffer.data(), size));
}
template <typename T>
void Engine<T>::transformOutput(std::vector<std::vector<std::vector<T>>> &input, std::vector<std::vector<T>> &output) {
+1 -1
View File
@@ -22,7 +22,7 @@ namespace ANSCENTER
{1005, "ANSOCR"},
{1006, "ANSALPR"},
{1007, "ANSCV"},
{1008, "ANSSRT"}
{1008, "ANSTS"}
};
ansllmLicenceValid = false;
for (const auto& [productId, productName] : licenseChecks) {
+84 -29
View File
@@ -11,12 +11,37 @@
namespace ANSCENTER {
// ====================================================================
// Sealed packages (.ansmodel)
//
// TensorRT still writes a plan to disk and still reads it back on the
// next run — that is what makes the second start fast, and a plan is a
// built artefact that carries no model. What must never be written is
// the ONNX itself, so the builder parses the decrypted bytes from
// memory and the path below names nothing that is ever created: it
// only tells the plan where to live and what to be called. The folder
// is the one the zip flow used, so a package converted from a zip
// reuses the plan that zip already built.
// ====================================================================
std::string ANSRTYOLO::PackagePlanBasePath(const std::string& packagePath) {
const std::string folder = GenerateOutputFolder(
packagePath, GetFileNameWithoutExtension(packagePath), false);
return CreateFilePath(folder, "train_last.onnx");
}
// ====================================================================
// ANSODBase interface — OptimizeModel
// ====================================================================
bool ANSRTYOLO::OptimizeModel(bool fp16, std::string& optimizedModelFolder) {
std::lock_guard<std::recursive_mutex> lock(_mutex);
if (!ANSODBase::OptimizeModel(fp16, optimizedModelFolder)) return false;
if (_fromPackage) {
// The plan is built from the decrypted bytes when the engine
// loads; there is no ONNX on disk to pre-optimise. Report the
// folder the plan is cached in so the caller sees a real path.
optimizedModelFolder = GetParentFolder(_modelFilePath);
return true;
}
if (!FileExist(_modelFilePath)) {
_logger.LogFatal("ANSRTYOLO::OptimizeModel", "Raw model file path does not exist", __FILE__, __LINE__);
return false;
@@ -96,21 +121,33 @@ namespace ANSCENTER {
m_options.engineFileDir = _modelFolder;
m_options.precision = (_fp16 ? Precision::FP16 : Precision::FP32);
_modelFilePath = CreateFilePath(_modelFolder, "train_last.onnx");
if (FileExist(_modelConfigFile)) {
ModelType modelType;
std::vector<int> inputShape;
_classes = ANSUtilityHelper::GetConfigFileContent(_modelConfigFile, modelType, inputShape);
if (inputShape.size() == 2) {
if (inputShape[0] > 0) _modelConfig.inpHeight = inputShape[0];
if (inputShape[1] > 0) _modelConfig.inpWidth = inputShape[1];
}
if (_fromPackage) {
// The package already carries the classes and the geometry,
// and there is no folder to look in. All that is needed here
// is where the built plan lives.
_modelFilePath = PackagePlanBasePath(modelZipFilePath);
m_options.engineFileDir = GetParentFolder(_modelFilePath);
// The defaults above have just overwritten the input size the
// package asked for; put it back.
ApplyPackageInputSize();
}
else {
_classFilePath = CreateFilePath(_modelFolder, "classes.names");
std::ifstream isValid(_classFilePath);
if (!isValid) LoadClassesFromString();
else LoadClassesFromFile();
_modelFilePath = CreateFilePath(_modelFolder, "train_last.onnx");
if (FileExist(_modelConfigFile)) {
ModelType modelType;
std::vector<int> inputShape;
_classes = ANSUtilityHelper::GetConfigFileContent(_modelConfigFile, modelType, inputShape);
if (inputShape.size() == 2) {
if (inputShape[0] > 0) _modelConfig.inpHeight = inputShape[0];
if (inputShape[1] > 0) _modelConfig.inpWidth = inputShape[1];
}
}
else {
_classFilePath = CreateFilePath(_modelFolder, "classes.names");
std::ifstream isValid(_classFilePath);
if (!isValid) LoadClassesFromString();
else LoadClassesFromFile();
}
}
if (this->_loadEngineOnCreation) {
@@ -119,7 +156,9 @@ namespace ANSCENTER {
static_cast<int>(m_options.precision), m_options.maxBatchSize };
m_trtEngine = EnginePoolManager<float>::instance().acquire(
m_poolKey, m_options, _modelFilePath,
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu);
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu,
_fromPackage ? _packageModel.data() : nullptr,
_fromPackage ? _packageModel.size() : 0);
m_usingSharedPool = (m_trtEngine != nullptr);
}
if (!m_trtEngine) {
@@ -221,7 +260,9 @@ namespace ANSCENTER {
static_cast<int>(m_options.precision), m_options.maxBatchSize };
m_trtEngine = EnginePoolManager<float>::instance().acquire(
m_poolKey, m_options, _modelFilePath,
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu);
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu,
_fromPackage ? _packageModel.data() : nullptr,
_fromPackage ? _packageModel.size() : 0);
m_usingSharedPool = (m_trtEngine != nullptr);
}
if (!m_trtEngine) {
@@ -295,21 +336,33 @@ namespace ANSCENTER {
m_options.engineFileDir = _modelFolder;
m_options.precision = (_fp16 ? Precision::FP16 : Precision::FP32);
_modelFilePath = CreateFilePath(_modelFolder, "train_last.onnx");
if (FileExist(_modelConfigFile)) {
ModelType modelType;
std::vector<int> inputShape;
_classes = ANSUtilityHelper::GetConfigFileContent(_modelConfigFile, modelType, inputShape);
if (inputShape.size() == 2) {
if (inputShape[0] > 0) _modelConfig.inpHeight = inputShape[0];
if (inputShape[1] > 0) _modelConfig.inpWidth = inputShape[1];
}
if (_fromPackage) {
// The package already carries the classes and the geometry,
// and there is no folder to look in. All that is needed here
// is where the built plan lives.
_modelFilePath = PackagePlanBasePath(modelZipFilePath);
m_options.engineFileDir = GetParentFolder(_modelFilePath);
// The defaults above have just overwritten the input size the
// package asked for; put it back.
ApplyPackageInputSize();
}
else {
_classFilePath = CreateFilePath(_modelFolder, "classes.names");
std::ifstream isValid(_classFilePath);
if (!isValid) LoadClassesFromString();
else LoadClassesFromFile();
_modelFilePath = CreateFilePath(_modelFolder, "train_last.onnx");
if (FileExist(_modelConfigFile)) {
ModelType modelType;
std::vector<int> inputShape;
_classes = ANSUtilityHelper::GetConfigFileContent(_modelConfigFile, modelType, inputShape);
if (inputShape.size() == 2) {
if (inputShape[0] > 0) _modelConfig.inpHeight = inputShape[0];
if (inputShape[1] > 0) _modelConfig.inpWidth = inputShape[1];
}
}
else {
_classFilePath = CreateFilePath(_modelFolder, "classes.names");
std::ifstream isValid(_classFilePath);
if (!isValid) LoadClassesFromString();
else LoadClassesFromFile();
}
}
labelMap.clear();
@@ -322,7 +375,9 @@ namespace ANSCENTER {
static_cast<int>(m_options.precision), m_options.maxBatchSize };
m_trtEngine = EnginePoolManager<float>::instance().acquire(
m_poolKey, m_options, _modelFilePath,
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu);
SUB_VALS, DIV_VALS, NORMALIZE, m_maxSlotsPerGpu,
_fromPackage ? _packageModel.data() : nullptr,
_fromPackage ? _packageModel.size() : 0);
m_usingSharedPool = (m_trtEngine != nullptr);
}
if (!m_trtEngine) {
+5
View File
@@ -52,6 +52,11 @@ namespace ANSCENTER {
~ANSRTYOLO();
private:
// Where a sealed package's TensorRT plan is cached, and what it is
// called. Nothing but the plan is ever written there — the model
// itself is parsed from memory.
static std::string PackagePlanBasePath(const std::string& packagePath);
std::string _modelFilePath;
bool _modelLoadValid{ false };
bool _fp16{ false };
+94
View File
@@ -552,6 +552,30 @@ namespace ANSCENTER {
const float* rawOutput = outputTensors[0].GetTensorMutableData<float>();
const std::vector<int64_t> outputShape = outputTensors[0].GetTensorTypeAndShapeInfo().GetShape();
if (outputShape.size() < 3) {
return detections;
}
// WHICH HEAD THIS MODEL HAS. An export with end2end on has already
// done its own non-maximum suppression and answers [1, N, 6] —
// rows of (x1 y1 x2 y2 score class) — where an ordinary YOLO
// answers [1, 4+nc, anchors]. Its backend marker is still
// tensorrt.txt either way, because the marker names the runtime
// and not the head, so the tensor is the only thing that knows.
//
// Reading one as the other does not fail: it takes 300 as a
// feature count, decides the model has 296 classes, reads
// coordinates as class scores and returns boxes covering the
// frame. The trap in the other direction is a two-class export,
// whose [1, 4+nc, N] is [1, 6, 8400] — a 6 in the shape and not a
// decoded head — so the 6 must be the LAST axis with a detection
// count before it: hundreds, never thousands of anchors.
if (outputShape[0] == 1 && outputShape[2] == 6 &&
outputShape[1] > 0 && outputShape[1] <= 1000) {
return postprocessEndToEnd(originalImageSize, resizedImageShape,
outputTensors, confThreshold);
}
// Determine the number of features and detections
const size_t num_features = outputShape[1];
const size_t num_detections = outputShape[2];
@@ -675,6 +699,76 @@ namespace ANSCENTER {
}
}
// An end-to-end head has already chosen its detections: the rows are boxes
// in model-pixel corners with a score and a class, and the suppression
// happened inside the graph. None is run here — running one anyway would
// hide a broken export instead of showing it.
std::vector<Object> YOLO12OD::postprocessEndToEnd(const cv::Size& originalImageSize,
const cv::Size& resizedImageShape,
std::vector<Ort::Value>& outputTensors,
float confThreshold)
{
std::vector<Object> detections;
if (outputTensors.empty()) return detections;
// GetTensorMutableData (not GetTensorData) on DML — the const variant
// hangs on AMD after ~8 calls. Read-only despite the name.
const float* rawOutput = outputTensors[0].GetTensorMutableData<float>();
const std::vector<int64_t> outputShape =
outputTensors[0].GetTensorTypeAndShapeInfo().GetShape();
if (outputShape.size() < 3) return detections;
const int numDets = static_cast<int>(outputShape[1]);
const int numFeat = static_cast<int>(outputShape[2]);
const int classNameSize = static_cast<int>(_classes.size());
detections.reserve(static_cast<size_t>(numDets));
for (int i = 0; i < numDets; ++i) {
const float* det = rawOutput + static_cast<size_t>(i) * numFeat;
const float conf = det[4];
// The rows are not obliged to be sorted, so every one is read.
if (conf <= confThreshold) continue;
// Corners in model pixels, handed to the same scaler the grid path
// uses: it undoes this engine's own letterbox and clips to the
// frame, so both heads agree about where a box is.
const float left = det[0];
const float top = det[1];
const float width = det[2] - det[0];
const float height = det[3] - det[1];
if (width <= 0.0f || height <= 0.0f) continue;
// BoundingBox is integral and these corners are not, so the
// rounding is spelled out rather than left to a narrowing
// conversion that would quietly truncate every box by up to a
// pixel.
const BoundingBox modelBox(static_cast<int>(std::lround(left)),
static_cast<int>(std::lround(top)),
static_cast<int>(std::lround(width)),
static_cast<int>(std::lround(height)));
const BoundingBox scaled =
scaleCoords(resizedImageShape, modelBox, originalImageSize, true);
if (scaled.width <= 0 || scaled.height <= 0) continue;
Object detection;
detection.confidence = conf;
detection.classId = static_cast<int>(det[5]);
detection.box = cv::Rect(scaled.x, scaled.y, scaled.width, scaled.height);
if (classNameSize > 0) {
detection.className =
(detection.classId >= 0 && detection.classId < classNameSize)
? _classes[detection.classId]
: _classes[classNameSize - 1];
} else {
detection.className = "Unknown";
}
detection.polygon = ANSUtilityHelper::RectToNormalizedPolygon(
detection.box, m_imgWidth, m_imgHeight);
detections.push_back(std::move(detection));
}
return detections;
}
std::vector<Object> YOLO12OD::detect(const cv::Mat& image, float confThreshold, float iouThreshold) {
std::lock_guard<std::recursive_mutex> lock(_mutex);
try {
+9
View File
@@ -59,6 +59,15 @@ namespace ANSCENTER {
std::vector<Ort::Value>& outputTensors,
float confThreshold, float iouThreshold);
/// The [1, N, 6] head: rows of (x1 y1 x2 y2 score class) whose
/// non-maximum suppression happened inside the graph. postprocess()
/// hands the tensor here when it recognises that shape; reading it
/// as a [1, 4+nc, anchors] grid returns boxes covering the frame.
std::vector<Object> postprocessEndToEnd(const cv::Size& originalImageSize,
const cv::Size& resizedImageShape,
std::vector<Ort::Value>& outputTensors,
float confThreshold);
private:
template <typename T>
typename std::enable_if<std::is_arithmetic<T>::value, T>::type
+13 -2
View File
@@ -8,6 +8,7 @@
#include <climits> // INT_MIN
#include "ANSLicense.h" // ANS_DBG macro for DebugView
#include "ANSODVendorGate.h" // ansod_vendor_gate::IsNvidiaGpuAvailable()
#include "ANSModelOpen.h" // modelpkg::IsPackagePath (sealed .ansmodel)
// Process-wide flag: when true, all engines force single-GPU path (no pool, no idle timers).
// Defined here, declared extern in EngineBuildLoadNetwork.inl.
@@ -1576,6 +1577,13 @@ ANSODENGINE_API int OptimizeModelStr(const char* modelFilePath, const char* mode
modelType, modelDetectionType, fp16, modelFilePath ? modelFilePath : "(null)");
try {
bool optimizedResult = false;
// A sealed .ansmodel is already the artefact: there is no zip to
// inflate and no loose ONNX to point an optimiser at. It still goes
// through the handle below, which opens it in memory and builds the
// TensorRT plan from the decrypted bytes — what it must not go
// through is ModelOptimizer, which only knows how to unzip.
const bool sealedPackage =
ANSCENTER::modelpkg::IsPackagePath(modelFilePath ? modelFilePath : "");
// NOTE: odMutex was removed here. OptimizeModelStr creates its own
// temporary Engine<float> on the stack — no shared state with running
// inference tasks. The global mutex was blocking ALL running tasks'
@@ -1628,7 +1636,7 @@ ANSODENGINE_API int OptimizeModelStr(const char* modelFilePath, const char* mode
}
if (modelType == 31) // If modelType is RTYOLO (31), handle separately.
if (modelType == 31 && !sealedPackage) // If modelType is RTYOLO (31), handle separately.
{
if (engineType == ANSCENTER::EngineType::NVIDIA_GPU) {
if (modelDetectionType == 0) {
@@ -1778,7 +1786,10 @@ ANSODENGINE_API int OptimizeModelStr(const char* modelFilePath, const char* mode
Handle.reset(); // Destroy engines BEFORE guard restores cache
if (optimizedResult && !modelFolder.empty()) return 1;
// A package reports the folder its plan was cached in; when the
// handle declined to name one there is still nothing wrong, because
// nothing was ever meant to be written.
if (optimizedResult && (!modelFolder.empty() || sealedPackage)) return 1;
else return 0;
}
catch (const std::exception& e) {
+17
View File
@@ -188,6 +188,17 @@ public:
/// Full path of the engine file resolved by the last buildLoadNetwork().
[[nodiscard]] const std::string& engineFilePath() const { return m_engineFilePath; }
// ── ONNX bytes held in memory (sealed .ansmodel packages) ───────────────
// A package is decrypted into memory and the plaintext model must never
// reach the disk, so the parser reads these bytes instead of opening
// onnxModelPath. The path is still passed in and still matters: it names
// the TensorRT plan that gets cached beside it, and a plan is a built
// artefact, not the model. Set the bytes before any build call; they are
// wiped by clearOnnxBytes() and by the destructor.
void setOnnxBytes(const void* data, size_t size);
void clearOnnxBytes();
[[nodiscard]] bool hasOnnxBytes() const { return !m_onnxBytes.empty(); }
/// Return the CUDA device index that preprocessing should target.
/// In pool mode this returns the device of the first alive slot;
/// in single-engine mode it returns m_options.deviceIndex.
@@ -244,6 +255,9 @@ public:
// retries until build succeeds or all candidates are exhausted.
// Fixed-spatial models fall through to a single build() call.
bool buildWithRetry(std::string onnxModelPath, const std::array<float, 3>& subVals, const std::array<float, 3>& divVals, bool normalize);
// Where the ONNX bytes come from: the in-memory buffer when one was set,
// otherwise the file at onnxModelPath.
bool readOnnxBytes(const std::string& onnxModelPath, std::vector<char>& buffer) const;
void getDeviceNames(std::vector<std::string>& deviceNames);
void clearGpuBuffers();
int getBindingIndexByName(const std::string& name);
@@ -401,6 +415,7 @@ private:
bool m_usingCachedEngine = false;
bool m_skipEngineCache = false; // true = bypass TRTEngineCache (for OptimizeModelStr)
bool m_skipOnnxRebuild = false; // true = don't delete/rebuild corrupt engine files (elastic growth)
std::vector<char> m_onnxBytes; // model bytes in memory (package); empty = read from disk
bool m_rebuiltFromOnnx = false; // last buildLoadNetwork built from ONNX (no matching cached engine)
std::string m_engineFilePath; // engine file resolved by last buildLoadNetwork
bool m_skipOnnxBuild = false; // true = skip ONNX→TRT build if no cached engine (demand growth)
@@ -575,6 +590,8 @@ template <typename T> Engine<T>::Engine() {
m_lastSetInputDims.clear();
}
template <typename T> Engine<T>::~Engine() {
// A decrypted model must not outlive the engine that parsed it.
try { clearOnnxBytes(); } catch (...) {}
// =========================================================================
// CRITICAL: Destructors must NEVER throw. Every CUDA/TRT call below is
// wrapped in try/catch so that a failure in one step does not skip the
+1 -1
View File
@@ -33,7 +33,7 @@ namespace ANSCENTER
{1005, "ANSOCR"},
{1006, "ANSALPR"},
{1007, "ANSCV"},
{1008, "ANSSRT"}
{1008, "ANSTS"}
};
ansawss3LicenceValid = false;
for (const auto& [productId, productName] : licenseChecks) {
+1 -1
View File
@@ -40,7 +40,7 @@ namespace ANSCENTER
{1005, "ANSOCR"},
{1006, "ANSALPR"},
{1007, "ANSCV"},
{1008, "ANSSRT"}
{1008, "ANSTS"}
};
ansutilityLicenceValid = false;
for (const auto& [productId, productName] : licenseChecks) {
@@ -328,12 +328,15 @@ static std::string pbDim(int64_t v) { // Dimension.dim_value
static std::string pbDynamicDim(const std::string& name) {
return pbField(1, pbField(2, name)); // dim_param: the axis is open
}
static std::string fakeOnnx(const std::string& dims) {
const std::string tensor = pbField(1, pbField(2, dims));
const std::string input = pbField(1, "images") + pbField(2, tensor);
const std::string graph = pbField(1, std::string(4096, 'n')) +
pbField(5, std::string(65536, 'w')) +
pbField(11, input);
static std::string fakeOnnx(const std::string& dims,
const std::string& outDims = std::string()) {
auto value = [](const std::string& name, const std::string& d) {
return pbField(1, name) + pbField(2, pbField(1, pbField(2, d)));
};
std::string graph = pbField(1, std::string(4096, 'n')) +
pbField(5, std::string(65536, 'w')) +
pbField(11, value("images", dims));
if (!outDims.empty()) graph += pbField(12, value("output0", outDims));
return pbField(7, graph);
}
@@ -405,6 +408,57 @@ static void testShapeFromTheGraph(const fs::path& dir) {
CHECK(modelpkg::OpenModelPackage(path, opened, err), ("open: " + err).c_str());
CHECK(opened.inputWidth == 224 && opened.inputHeight == 224,
"so the engine is told 224 from the graph, not left to assume 640");
// AN END-TO-END HEAD IN A PACKAGE THAT SAYS "yolo". Boxes used to take the
// family from the backend marker, and the marker names the runtime, not the
// head — so a yolo26 export with end2end on arrived declared as plain yolo.
// Decoding [1,N,6] as a v8 grid reads coordinates as confidences: boxes
// everywhere, and no error to notice. The graph settles it.
{
const std::string graph =
fakeOnnx(pbDim(1) + pbDim(3) + pbDim(640) + pbDim(640),
pbDim(1) + pbDim(300) + pbDim(6));
modelpkg::PackRequest r = req;
r.manifest = Json{{"task", "detection"},
{"family", "yolo"},
{"classes", Json::array({Json{{"id", 1}, {"name", "Tooth"}}})},
{"models", Json::array({Json{{"id", "m0"},
{"payload", "p0"},
{"entry", true}}})}};
r.models.clear();
r.models.push_back({"p0", std::vector<uint8_t>(graph.begin(), graph.end())});
const std::string p = (dir / "e2e.ansmodel").string();
CHECK(modelpkg::pack(p, r, err), ("pack e2e: " + err).c_str());
modelpkg::OpenedModel m;
CHECK(modelpkg::OpenModelPackage(p, m, err) && m.family == "yolo10",
"an end-to-end head is read as yolo10 whatever the package says");
CHECK(m.warning.find("end-to-end") != std::string::npos,
"and the reason is said out loud rather than silently swapped");
CHECK(m.legacyMarker.empty() || m.legacyMarker == "tensorrt.txt",
"while the legacy marker is left as it was");
}
// A two-class v8 export is [1, 6, 8400]: a 6 in the shape, not a head that
// has already decoded itself.
{
const std::string graph =
fakeOnnx(pbDim(1) + pbDim(3) + pbDim(640) + pbDim(640),
pbDim(1) + pbDim(6) + pbDim(8400));
modelpkg::PackRequest r = req;
r.manifest = Json{{"task", "detection"},
{"family", "yolo"},
{"classes", Json::array({Json{{"id", 1}, {"name", "Tooth"}}})},
{"models", Json::array({Json{{"id", "m0"},
{"payload", "p0"},
{"entry", true}}})}};
r.models.clear();
r.models.push_back({"p0", std::vector<uint8_t>(graph.begin(), graph.end())});
const std::string p = (dir / "twoclass.ansmodel").string();
CHECK(modelpkg::pack(p, r, err), ("pack v8: " + err).c_str());
modelpkg::OpenedModel m;
CHECK(modelpkg::OpenModelPackage(p, m, err) && m.family == "yolo",
"a 6 on the wrong axis with thousands of anchors stays plain yolo");
}
}
int main(int argc, char** argv) {
@@ -2,6 +2,8 @@
#include "ANSCUSTOMPY.h"
#include "ANSMotionDetector.h"
#include "ANSONNXSAM3.h"
#include <cstdlib>
#include <iostream>
#include <thread>
#include <atomic>
#include <chrono>
@@ -2235,8 +2237,31 @@ int FireNSmokeTest() {
return 0;
}
int main()
// RUN A MODEL THIS MACHINE ACTUALLY HAS. Every test below names a path from
// whichever workstation it was written on, so on any other machine the harness
// stops at "Model zip file is not exist" before it ever reaches the GPU. Given
// arguments it runs what it is told instead:
//
// ANSODEngine-UnitTest.exe <model> <video> [engineType] [detectionType]
//
// `model` is a .zip or a sealed .ansmodel — the engine opens either, and a
// package is the only way to see a sealed model run on the GPU. engineType is
// the ModelType enum (3 auto-detects a YOLO export, 30 ONNX YOLO, 16 custom,
// 32 RTMPose); detectionType 1 is object detection.
//
// With no arguments it does exactly what it did before.
int main(int argc, char** argv)
{
if (argc >= 3) {
const std::string model = argv[1];
const std::string video = argv[2];
const int engineType = (argc > 3) ? std::atoi(argv[3]) : 3;
const int detectionType = (argc > 4) ? std::atoi(argv[4]) : 1;
std::cout << "model: " << model << "\nvideo: " << video
<< "\nengineType: " << engineType
<< ", detectionType: " << detectionType << std::endl;
return VideoDetectorEngine_CPP(model, video, engineType, detectionType);
}
//FallDetectorTest();
//ALPROD_Test();