19 KiB
AMD DirectML FP16 Converter — Implementation Plan
Status: Not started. Pick this up in a future session. Estimated effort: ~1 week of focused work. Owner of context: This document is self-contained — you don't need the prior conversation to execute it.
1. Goal
Recover ~30–50% of per-call inference latency on AMD Radeon iGPUs (DML EP)
by running models in FP16 instead of FP32. Pure C++ implementation —
no Python tooling at any stage. Mirror the existing TensorRT pattern
where OrtHandlerOptions::trtFP16 = true causes the engine layer to
build/cache an FP16 plan automatically at session-create time.
The current AMD performance baseline (after the always-serial-on-AMD fix) is ~5.4 fps for full-frame ALPR. Target after FP16: 7–8 fps.
2. Why C++ (and not the standard Python toolchain)
The standard tool for this is onnxconverter-common.float16.convert_float_to_float16.
Project rule disallows Python anywhere in the workflow, so we must
re-implement the algorithm in C++ inside ANSCORE.
The DirectML EP itself has no dml_fp16_enable runtime knob (unlike TRT's
trt_fp16_enable). DML runs whatever precision the ONNX model declares.
So the C++ work has to produce a transformed FP16 ONNX, then hand that
to the DML EP.
3. Architecture (mirrors the TRT pattern)
TRT path (existing):
trtFP16 = true → ORT TRT EP options → TRT engine builder rewrites
→ engine cached on disk → inference
DML path (new):
dmlFP16 = true → ANSCORE C++ rewrite → cached as <hash>.fp16.onnx
→ DML EP loads cached FP16 model → inference
Cache invalidation, per-model granularity, and the "first run is slow" semantics all match TRT. From the call-site perspective the TRT and DML options are symmetric.
4. Current state of the codebase (relevant files)
These files already exist and are the integration points. Don't recreate them; modify them.
| File | Relevant content | What this plan adds |
|---|---|---|
engines/ONNXEngine/ONNXEngine.h |
OrtHandlerOptions struct. AMD section commented "OPEN FOR OPTIMIZATION" — empty today |
Add dmlFP16, dmlOptimizedModelDir, dmlFP16OpBlockList fields |
engines/ONNXEngine/ONNXEngine.cpp |
BasicOrtHandler::initialize_handler() — DML EP branch (search for EngineType::AMD_GPU) |
Insert path-resolution call before session creation |
engines/ONNXEngine/ONNXEngine.cpp |
Already houses DmlSerialState + DmlInferenceGuard (the always-serial-on-AMD fix from previous work) |
Reuse the same TU for FP16 helpers |
engines/ONNXEngine/Fp16GraphRewriter.{h,cpp} |
Does not exist yet | Create — main implementation of this plan |
modules/ANSOCR/ANSONNXOCR/PaddleOCRV5Engine.cpp |
BuildXxxOcrOptions() helpers populating per-model OrtHandlerOptions |
Add o.dmlFP16 = true/false per model |
Caller code in ANSONNXYOLO, ONNXOCRDetector, ONNXOCRRecognizer |
Already constructs sessions via BasicOrtHandler |
No changes — the FP16 swap is invisible to callers |
5. Step-by-step plan
Step 1 — Add AMD knobs to OrtHandlerOptions (half day)
In engines/ONNXEngine/ONNXEngine.h, replace the AMD placeholder:
// ----------------------------------------------------------------
// AMD (DirectML EP) — FP16 graph rewrite
// ----------------------------------------------------------------
// Convert FP32 weights to FP16 before session creation. Mirrors
// trtFP16 semantically: an in-process graph rewrite produces an FP16
// ONNX, cached on disk under dmlOptimizedModelDir, and the DML EP
// loads the cached FP16 model. External inputs/outputs stay FP32 via
// Cast nodes inserted at the boundaries — caller code is unchanged.
// Ignored on every non-DML EP. Default off until per-model accuracy
// validation completes.
bool dmlFP16 = false;
// Where to cache rewritten FP16 ONNX models. Empty → default
// %TEMP%/ANSCENTER/DMLOptimizedCache. Cache key is a hash of
// (source ONNX bytes + dmlFP16 flag + opBlockList). Stale entries
// are silently replaced when source changes.
std::string dmlOptimizedModelDir;
// Op types whose outputs MUST remain FP32 (NMS, TopK, RoiAlign, plus
// any op observed to produce NaN/Inf in FP16 for THIS model). Empty
// = use the built-in default block list (see kDefaultFp16BlockList in
// Fp16GraphRewriter.cpp). Per-model override.
std::vector<std::string> dmlFP16OpBlockList;
Step 2 — Add ONNX protobuf to the build (half day)
You need the onnx.proto3 schema compiled to C++ via protoc. Two
sub-options:
Option 2A (recommended): vendor the proto file.
- Download
onnx.proto3from https://github.com/onnx/onnx (current release tag). Put it atengines/ONNXEngine/proto/onnx.proto3. - Add a CMake
add_custom_commandcallingprotoc --cpp_out=...at configure time, generatingonnx.pb.{h,cc}into the build directory. - Link
Fp16GraphRewriteragainstprotobuf::libprotobuf.
libprotobuf is almost certainly already a transitive dep of
onnxruntime — confirm by checking
C:\ANSLibs\onnxruntime\include for google/protobuf/... headers and
C:\ANSLibs\protobuf (or vcpkg's protobuf install). Just expose the
include directories to your new target.
Option 2B: vendor the full ONNX C++ library. Bigger dep but provides typed helpers, validators, and a checker. Skip unless 2A's hand-rolled approach blows up on edge cases.
Step 3 — Implement Fp16GraphRewriter (~2 days for core)
Create engines/ONNXEngine/Fp16GraphRewriter.h:
#pragma once
#include <string>
#include <vector>
#include <unordered_set>
namespace ANSCENTER {
// Returns the path to an FP16 version of fp32Path. If cacheDir already
// contains a fresh FP16 cache for this model, returns the cached path
// directly (fast path). Otherwise loads fp32Path, runs the in-memory
// FP16 graph rewrite, writes to cacheDir, and returns the new path.
//
// Cache filename embeds a hash of (fp32 file content + opBlockList) so
// a model update or blocklist change automatically invalidates the
// cache — same pattern as TRT engine cache invalidation.
//
// On any failure (parse error, write error, etc.) returns fp32Path
// unchanged and logs a warning via ANS_DBG. The session creation
// continues with the FP32 model — this function NEVER blocks the
// inference path.
std::string ResolveFp16OnnxPath(
const std::string& fp32Path,
const std::string& cacheDir,
const std::vector<std::string>& opBlockList);
} // namespace ANSCENTER
Create engines/ONNXEngine/Fp16GraphRewriter.cpp. Key components:
3a. IEEE-754 binary32 → binary16 helper
Standard 30-line implementation, round-to-nearest-even, handle NaN/Inf.
Reference: numpy's numpy/core/src/npymath/halffloat.c or
Eigen/src/Core/arch/Default/Half.h. Self-contained, no deps.
3b. Default op blocklist
static const std::unordered_set<std::string> kDefaultFp16BlockList = {
// Object-detection post-processing — known FP16-unsafe
"NonMaxSuppression", "TopK", "RoiAlign",
// Classical ML ops — never have FP16 implementations
"ArrayFeatureExtractor", "Binarizer", "CastMap",
"CategoryMapper", "DictVectorizer", "FeatureVectorizer",
"Imputer", "LabelEncoder", "LinearClassifier",
"LinearRegressor", "Normalizer", "OneHotEncoder",
"RandomUniform", "SVMClassifier", "SVMRegressor",
"Scaler", "TreeEnsembleClassifier", "TreeEnsembleRegressor",
"ZipMap",
};
3c. Initializer conversion
Walk model.graph().initializer(). For every TensorProto with
data_type == FLOAT:
- Pull the float data (it lives in
raw_dataORfloat_data— handle both). - Convert each float32 to uint16 binary16.
- Set
data_type = FLOAT16, clearfloat_data, write the new bytes toraw_data.
Watch for: external data references (when data_location == EXTERNAL) — for the YOLO and OCR models in this project, weights are
embedded, but write a guard: if external, skip with a warning.
3d. Cast-node insertion at boundaries
For each external input (entries in model.graph().input() that are
NOT initializer names):
- Note the input's tensor name
orig. - Append a new
NodePrototo the graph:Cast(input=orig, output=orig + "_fp16", to=FLOAT16). - Walk every existing node in the graph; for any input that equals
orig, replace it withorig + "_fp16". Be careful not to rewire the new Cast node itself (skip the last node added).
For each external output: insert Cast(FP16 → FP32) symmetrically.
3e. value_info type updates
For every ValueInfoProto in graph().value_info():
- If the tensor is the input to OR output of a block-listed op, leave
its
tensor_type.elem_typeas FP32. - Otherwise update to FP16.
This step is the most error-prone in practice. When in doubt, leave elem_type alone — the runtime will accept inferred types. Test correctness empirically with the validation harness in Step 7.
3f. Subgraphs
If/Loop/Scan ops have nested subgraphs (AttributeProto of type
GRAPH). Recurse into each. The OCR recognizer doesn't have these;
neither does YOLO post-Ultralytics-export — but write the recursion
defensively, otherwise this fails silently on future models.
3g. Cache management
Cache filename: <model_basename>.<sha1_8>.fp16.onnx, where sha1_8 is
the first 8 hex chars of SHA1(fp32_file_bytes || serialized opBlockList || "v1"). The "v1" string lets you bust the cache for
all models if you fix a converter bug later.
Cache directory: create on demand with std::filesystem::create_directories.
Default: std::filesystem::temp_directory_path() / "ANSCENTER" / "DMLOptimizedCache".
Lookup: if cache file exists AND its mtime is newer than the source ONNX, use it. Else regenerate.
Step 4 — Wire into initialize_handler (half day)
In engines/ONNXEngine/ONNXEngine.cpp, find the DML branch in
BasicOrtHandler::initialize_handler(). Insert path resolution before
session creation:
case EngineType::AMD_GPU: {
std::string onnxToLoad = _onnx_path;
if (options.dmlFP16) {
const std::string cacheDir =
options.dmlOptimizedModelDir.empty()
? GetDefaultDmlCacheDir()
: options.dmlOptimizedModelDir;
onnxToLoad = ResolveFp16OnnxPath(
_onnx_path, cacheDir, options.dmlFP16OpBlockList);
ANS_DBG("OrtHandler",
"AMD FP16: source=%s cached=%s",
_onnx_path.c_str(), onnxToLoad.c_str());
}
if (!TryAppendDirectML(opts)) { /* existing fallback */ }
ort_session = std::make_unique<Ort::Session>(
env, onnxToLoad.c_str(), opts);
break;
}
GetDefaultDmlCacheDir() is a small helper in Fp16GraphRewriter.cpp.
Step 5 — Per-model configuration (1 day)
Update the option-builder functions to opt models into FP16:
modules/ANSOCR/ANSONNXOCR/PaddleOCRV5Engine.cpp (and equivalents):
OrtHandlerOptions BuildLpDetectorOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
o.dmlFP16 = true; // YOLO det validated FP16-safe
return o;
}
OrtHandlerOptions BuildOcrDetectorOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
o.dmlFP16 = true; // DBNet text-det validated
return o;
}
OrtHandlerOptions BuildOcrRecognizerOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
// OCR recognizer is FP16-sensitive on small text — keep AMD FP32
// until per-plate accuracy validation passes (see Step 7).
o.dmlFP16 = false;
return o;
}
Helmet OD (in ANSCUSTOMDetector model loading) — set dmlFP16 = true.
Step 6 — DebugView logging (per MEMORY.md rule)
Sprinkle ANS_DBG markers at every decision/error point in
Fp16GraphRewriter.cpp:
- Entry:
ANS_DBG("Fp16Rewriter", "ResolveFp16OnnxPath: src=%s cache=%s", fp32Path.c_str(), cacheDir.c_str()); - Cache hit:
ANS_DBG("Fp16Rewriter", "cache HIT %s", cachedPath.c_str()); - Cache miss → rewrite:
ANS_DBG("Fp16Rewriter", "cache MISS — rewriting (initializers=%zu nodes=%zu)", ...); - Per phase: log initializer-conversion count, Cast-insertion count, value_info update count.
- On error:
ANS_DBG("Fp16Rewriter", "ERROR: %s — falling back to FP32 source", ex.what());and return the original FP32 path (never block inference).
Step 7 — Per-model accuracy validation (1–2 days)
You can't ship FP16 without per-model accuracy validation. Build a small C++ harness — separate executable target, runs against labeled fixtures from your test data:
tools/AccuracyComparator/AccuracyComparator.cpp
- Loads two sessions (FP32 and FP16) on CPU EP for determinism.
- Feeds N representative frames through both.
- For YOLO det: compare box IoU, score deltas, count of boxes lost.
- For OCR det: compare text-region masks (pixel-wise IoU on the prob
map binary at threshold).
- For OCR rec: compare DECODED CTC strings — not raw logits.
A single character delta on a plate is a regression.
- Emit pass/fail against a configurable tolerance.
Acceptance criteria:
| Model | Tolerance | What "pass" means |
|---|---|---|
| YOLO LP detector | mean IoU ≥ 0.95 vs FP32 boxes | No boxes lost, score delta < 0.05 |
| YOLO Helmet OD | mean IoU ≥ 0.95 | Same |
| OCR DBNet | mask IoU ≥ 0.92 at threshold | No new false negatives |
| OCR CTC recognizer | char accuracy within 0.5% | If fails, leave AMD FP32 |
Run on at least 500 frames per model from production data.
Step 8 — Build, deploy, measure (half day)
Build artifacts: ANSODEngine.dll, ANSOCR.dll. No new DLL — the
rewriter lives inside the existing engine layer.
Deploy to AMD test box. First run: each session takes an extra
2–10 s during initial cache build. Look for [Fp16Rewriter] cache MISS — rewriting lines.
Subsequent runs: instant cache hit. Look for [Fp16Rewriter] cache HIT.
Measure:
- Per-call DML latency in
STATSlines (mean, p99). - Total fps over a 60-second run.
- Compare to the always-serial-on-AMD baseline (5.4 fps full-frame ALPR).
- Target: ≥ 7 fps.
If a model regresses accuracy in the field (rare with the validation
harness in Step 7, but possible), flip its dmlFP16 back to false
and ship a hotfix. No code change needed beyond the option flag.
6. Dependencies summary
| Dep | Where it lives | New or existing |
|---|---|---|
libprotobuf (C++ runtime) |
Almost certainly already vendored via onnxruntime; otherwise add via vcpkg or C:\ANSLibs\protobuf |
Existing, just expose includes |
onnx.proto3 schema file |
Vendor at engines/ONNXEngine/proto/onnx.proto3 from ONNX repo |
New, ~1 KB file |
protoc compiler |
Comes with libprotobuf install | Existing, just call from CMake |
<filesystem> for cache dir |
C++17 standard | Existing |
| SHA1 helper for cache key | Pull from any tiny public-domain impl, or use libcrypto if linked | Tiny new file (~80 lines) |
No third-party FP16 libraries. No Python. No new DLLs in deployment.
7. Risks and mitigations
| Risk | Likelihood | Mitigation |
|---|---|---|
| Conversion produces NaN/Inf in some op | Medium | Default blocklist + per-model dmlFP16OpBlockList override. Always validate with Step 7 harness before flipping a model on. |
| Some FP16 op has no DML kernel → falls back to CPU silently → SLOWER than FP32 | Medium | Profile each model on AMD with ORT's profiling JSON output (SessionOptions::EnableProfiling). If a CPU fallback shows up, blocklist the offending op for that model. |
| ONNX schema version mismatch (we vendor proto3, models use proto with newer fields) | Low | Models from this project are exported by Ultralytics + PaddleOCR — both target stable opset 12-17 ranges that proto3 covers fully. Validate by attempting parse on a known-good model during Step 3. |
| Cache directory permissions on customer machines | Low | Default to %TEMP% which is always writable; allow override via dmlOptimizedModelDir. On any cache-write failure, fall back to FP32 source path (no inference impact). |
| Subgraph (If/Loop/Scan) bug | Low for current models | Recurse defensively in Step 3f; assert if a subgraph is encountered, fail-safe to FP32. |
8. Out of scope (do NOT do as part of this plan)
- Auto mixed precision (per-op accuracy-driven precision selection). That requires running a representative dataset through the model to decide which ops can safely be FP16. Doable in C++ but adds 1+ week. Defer to a follow-up plan.
- Model quantization to INT8. Different problem, different tooling (calibration data needed).
- Switching from DML to MIGraphX or ROCm. Different EP, different port. Defer.
- Recovering depth=2 parallelism on AMD. The always-serial fix is in place and stable; this plan is purely about per-call speedup while remaining serial. Combine the two for net throughput gain.
9. Project rules to follow (from MEMORY.md)
- Edit directly in main checkout. No worktree, no branch.
- No commits without explicit user request. When this plan ships, don't auto-commit.
- Sprinkle
ANS_DBGon new code paths — entry, decisions, errors. See Step 6 for exact placement.
10. Acceptance checklist
When this plan is done, the following must all be true:
OrtHandlerOptions::dmlFP16exists and is documented inline.Fp16GraphRewriter.{h,cpp}exists withResolveFp16OnnxPathexposed.BasicOrtHandler::initialize_handlercalls the resolver in the AMD branch whendmlFP16is set.- At least one production model (LP YOLO det) is opted in via its option-builder.
- Accuracy harness in
tools/AccuracyComparator/exists and produces pass/fail for the opted-in models. - AMD test-box measurement shows per-call DML latency reduced by ≥ 25% on the opted-in model vs FP32 baseline.
- No regression on Intel or NVIDIA — the FP16 path never executes
on those EPs (
if (options.dmlFP16)is gated onEngineType::AMD_GPUvia the case-branch placement). - DebugView logs cleanly: cache HIT / cache MISS / rewrite phases / errors all visible.
- First-run cache-build latency is ≤ 10 s per model on AMD test box.
- Subsequent-run cache-load latency is ≤ 100 ms per model.
11. Open questions to resolve at start
When you start this plan in a future session, answer these first:
- Is
libprotobufalready linked into ANSODEngine.dll (transitively via onnxruntime)? Rundumpbin /imports ANSODEngine.dll | grep protobufto confirm. If yes, skip the vcpkg/manual install step. - Where does
C:\ANSLibs\onnxruntime\includelive exactly? Confirm path for the protoc include directive. - Which ONNX opset version do your YOLO and OCR models target?
Run
Ort::Sessionwith profiling once and check the model metadata. This confirms whichonnx.proto3revision to vendor. - Is the initial AMD measurement still ~5.4 fps with the always-serial fix in place? If something regressed in the meantime, fix that first before adding FP16.
Last updated: when this plan was authored. No code shipped yet.