fix(YOLO12OD): decode an end-to-end head instead of reading it as a grid

This engine took the feature count from outputShape[1] unconditionally.
For a yolo26 export with end2end on — [1, 300, 6], rows of
(x1 y1 x2 y2 score class) with the suppression already inside the graph
— that means 296 classes, coordinates read as class scores, and boxes
covering the frame. No error, just nonsense, which is exactly what the
Labelling Tool produced from the same model before its decoder was
fixed.

Its sibling ANSONNXYOLO has recognised that head from the tensor shape
for a while; this one had not, and the backend marker cannot tell them
apart because it names the runtime rather than the head.

The branch fires only where the tensor cannot be a grid: the 6 must be
the last axis with a detection count before it, hundreds rather than the
thousands of anchors a two-class [1, 6, 8400] export carries.

The decoder reuses this engine's own scaleCoords rather than repeating
the letterbox arithmetic, so both heads undo the padding the same way
and agree about where a box is, and the rounding into the integral box
type is spelled out instead of left to a narrowing conversion.

NOT COMPILED. The engine layer needs CUDA and the ANSLibs tree, neither
of which exists on this machine, so this is verified by reading: the
shape rule matches the one the other three products now share, every
symbol it uses is declared in this translation unit, and the arithmetic
is delegated to the function the grid path already trusts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-13 06:55:45 +10:00
co-authored by Claude Opus 5
parent ea877a61bc
commit 3798e3513b
2 changed files with 103 additions and 0 deletions
+94
View File
@@ -552,6 +552,30 @@ namespace ANSCENTER {
const float* rawOutput = outputTensors[0].GetTensorMutableData<float>();
const std::vector<int64_t> outputShape = outputTensors[0].GetTensorTypeAndShapeInfo().GetShape();
if (outputShape.size() < 3) {
return detections;
}
// WHICH HEAD THIS MODEL HAS. An export with end2end on has already
// done its own non-maximum suppression and answers [1, N, 6] —
// rows of (x1 y1 x2 y2 score class) — where an ordinary YOLO
// answers [1, 4+nc, anchors]. Its backend marker is still
// tensorrt.txt either way, because the marker names the runtime
// and not the head, so the tensor is the only thing that knows.
//
// Reading one as the other does not fail: it takes 300 as a
// feature count, decides the model has 296 classes, reads
// coordinates as class scores and returns boxes covering the
// frame. The trap in the other direction is a two-class export,
// whose [1, 4+nc, N] is [1, 6, 8400] — a 6 in the shape and not a
// decoded head — so the 6 must be the LAST axis with a detection
// count before it: hundreds, never thousands of anchors.
if (outputShape[0] == 1 && outputShape[2] == 6 &&
outputShape[1] > 0 && outputShape[1] <= 1000) {
return postprocessEndToEnd(originalImageSize, resizedImageShape,
outputTensors, confThreshold);
}
// Determine the number of features and detections
const size_t num_features = outputShape[1];
const size_t num_detections = outputShape[2];
@@ -675,6 +699,76 @@ namespace ANSCENTER {
}
}
// An end-to-end head has already chosen its detections: the rows are boxes
// in model-pixel corners with a score and a class, and the suppression
// happened inside the graph. None is run here — running one anyway would
// hide a broken export instead of showing it.
std::vector<Object> YOLO12OD::postprocessEndToEnd(const cv::Size& originalImageSize,
const cv::Size& resizedImageShape,
std::vector<Ort::Value>& outputTensors,
float confThreshold)
{
std::vector<Object> detections;
if (outputTensors.empty()) return detections;
// GetTensorMutableData (not GetTensorData) on DML — the const variant
// hangs on AMD after ~8 calls. Read-only despite the name.
const float* rawOutput = outputTensors[0].GetTensorMutableData<float>();
const std::vector<int64_t> outputShape =
outputTensors[0].GetTensorTypeAndShapeInfo().GetShape();
if (outputShape.size() < 3) return detections;
const int numDets = static_cast<int>(outputShape[1]);
const int numFeat = static_cast<int>(outputShape[2]);
const int classNameSize = static_cast<int>(_classes.size());
detections.reserve(static_cast<size_t>(numDets));
for (int i = 0; i < numDets; ++i) {
const float* det = rawOutput + static_cast<size_t>(i) * numFeat;
const float conf = det[4];
// The rows are not obliged to be sorted, so every one is read.
if (conf <= confThreshold) continue;
// Corners in model pixels, handed to the same scaler the grid path
// uses: it undoes this engine's own letterbox and clips to the
// frame, so both heads agree about where a box is.
const float left = det[0];
const float top = det[1];
const float width = det[2] - det[0];
const float height = det[3] - det[1];
if (width <= 0.0f || height <= 0.0f) continue;
// BoundingBox is integral and these corners are not, so the
// rounding is spelled out rather than left to a narrowing
// conversion that would quietly truncate every box by up to a
// pixel.
const BoundingBox modelBox(static_cast<int>(std::lround(left)),
static_cast<int>(std::lround(top)),
static_cast<int>(std::lround(width)),
static_cast<int>(std::lround(height)));
const BoundingBox scaled =
scaleCoords(resizedImageShape, modelBox, originalImageSize, true);
if (scaled.width <= 0 || scaled.height <= 0) continue;
Object detection;
detection.confidence = conf;
detection.classId = static_cast<int>(det[5]);
detection.box = cv::Rect(scaled.x, scaled.y, scaled.width, scaled.height);
if (classNameSize > 0) {
detection.className =
(detection.classId >= 0 && detection.classId < classNameSize)
? _classes[detection.classId]
: _classes[classNameSize - 1];
} else {
detection.className = "Unknown";
}
detection.polygon = ANSUtilityHelper::RectToNormalizedPolygon(
detection.box, m_imgWidth, m_imgHeight);
detections.push_back(std::move(detection));
}
return detections;
}
std::vector<Object> YOLO12OD::detect(const cv::Mat& image, float confThreshold, float iouThreshold) {
std::lock_guard<std::recursive_mutex> lock(_mutex);
try {
+9
View File
@@ -59,6 +59,15 @@ namespace ANSCENTER {
std::vector<Ort::Value>& outputTensors,
float confThreshold, float iouThreshold);
/// The [1, N, 6] head: rows of (x1 y1 x2 y2 score class) whose
/// non-maximum suppression happened inside the graph. postprocess()
/// hands the tensor here when it recognises that shape; reading it
/// as a [1, 4+nc, anchors] grid returns boxes covering the frame.
std::vector<Object> postprocessEndToEnd(const cv::Size& originalImageSize,
const cv::Size& resizedImageShape,
std::vector<Ort::Value>& outputTensors,
float confThreshold);
private:
template <typename T>
typename std::enable_if<std::is_arithmetic<T>::value, T>::type