Files
ANSCORE/AMD_DML_FP16_Converter_Plan.md
2026-05-01 11:16:47 +10:00

453 lines
19 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AMD DirectML FP16 Converter — Implementation Plan
**Status:** Not started. Pick this up in a future session.
**Estimated effort:** ~1 week of focused work.
**Owner of context:** This document is self-contained — you don't need the
prior conversation to execute it.
---
## 1. Goal
Recover ~30–50% of per-call inference latency on AMD Radeon iGPUs (DML EP)
by running models in FP16 instead of FP32. Pure C++ implementation —
no Python tooling at any stage. Mirror the existing TensorRT pattern
where `OrtHandlerOptions::trtFP16 = true` causes the engine layer to
build/cache an FP16 plan automatically at session-create time.
The current AMD performance baseline (after the always-serial-on-AMD fix)
is ~5.4 fps for full-frame ALPR. Target after FP16: 7–8 fps.
## 2. Why C++ (and not the standard Python toolchain)
The standard tool for this is `onnxconverter-common.float16.convert_float_to_float16`.
Project rule disallows Python anywhere in the workflow, so we must
re-implement the algorithm in C++ inside ANSCORE.
The DirectML EP itself has no `dml_fp16_enable` runtime knob (unlike TRT's
`trt_fp16_enable`). DML runs whatever precision the ONNX model declares.
So the C++ work has to produce a transformed FP16 ONNX, then hand that
to the DML EP.
## 3. Architecture (mirrors the TRT pattern)
```
TRT path (existing):
trtFP16 = true → ORT TRT EP options → TRT engine builder rewrites
→ engine cached on disk → inference
DML path (new):
dmlFP16 = true → ANSCORE C++ rewrite → cached as <hash>.fp16.onnx
→ DML EP loads cached FP16 model → inference
```
Cache invalidation, per-model granularity, and the "first run is slow"
semantics all match TRT. From the call-site perspective the TRT and DML
options are symmetric.
## 4. Current state of the codebase (relevant files)
These files already exist and are the integration points. Don't recreate
them; modify them.
| File | Relevant content | What this plan adds |
|---|---|---|
| `engines/ONNXEngine/ONNXEngine.h` | `OrtHandlerOptions` struct. AMD section commented "OPEN FOR OPTIMIZATION" — empty today | Add `dmlFP16`, `dmlOptimizedModelDir`, `dmlFP16OpBlockList` fields |
| `engines/ONNXEngine/ONNXEngine.cpp` | `BasicOrtHandler::initialize_handler()` — DML EP branch (search for `EngineType::AMD_GPU`) | Insert path-resolution call before session creation |
| `engines/ONNXEngine/ONNXEngine.cpp` | Already houses `DmlSerialState` + `DmlInferenceGuard` (the always-serial-on-AMD fix from previous work) | Reuse the same TU for FP16 helpers |
| `engines/ONNXEngine/Fp16GraphRewriter.{h,cpp}` | **Does not exist yet** | Create — main implementation of this plan |
| `modules/ANSOCR/ANSONNXOCR/PaddleOCRV5Engine.cpp` | `BuildXxxOcrOptions()` helpers populating per-model `OrtHandlerOptions` | Add `o.dmlFP16 = true/false` per model |
| Caller code in `ANSONNXYOLO`, `ONNXOCRDetector`, `ONNXOCRRecognizer` | Already constructs sessions via `BasicOrtHandler` | No changes — the FP16 swap is invisible to callers |
## 5. Step-by-step plan
### Step 1 — Add AMD knobs to `OrtHandlerOptions` (half day)
In `engines/ONNXEngine/ONNXEngine.h`, replace the AMD placeholder:
```cpp
// ----------------------------------------------------------------
// AMD (DirectML EP) — FP16 graph rewrite
// ----------------------------------------------------------------
// Convert FP32 weights to FP16 before session creation. Mirrors
// trtFP16 semantically: an in-process graph rewrite produces an FP16
// ONNX, cached on disk under dmlOptimizedModelDir, and the DML EP
// loads the cached FP16 model. External inputs/outputs stay FP32 via
// Cast nodes inserted at the boundaries — caller code is unchanged.
// Ignored on every non-DML EP. Default off until per-model accuracy
// validation completes.
bool dmlFP16 = false;
// Where to cache rewritten FP16 ONNX models. Empty → default
// %TEMP%/ANSCENTER/DMLOptimizedCache. Cache key is a hash of
// (source ONNX bytes + dmlFP16 flag + opBlockList). Stale entries
// are silently replaced when source changes.
std::string dmlOptimizedModelDir;
// Op types whose outputs MUST remain FP32 (NMS, TopK, RoiAlign, plus
// any op observed to produce NaN/Inf in FP16 for THIS model). Empty
// = use the built-in default block list (see kDefaultFp16BlockList in
// Fp16GraphRewriter.cpp). Per-model override.
std::vector<std::string> dmlFP16OpBlockList;
```
### Step 2 — Add ONNX protobuf to the build (half day)
You need the `onnx.proto3` schema compiled to C++ via `protoc`. Two
sub-options:
**Option 2A (recommended): vendor the proto file.**
1. Download `onnx.proto3` from https://github.com/onnx/onnx (current
release tag). Put it at `engines/ONNXEngine/proto/onnx.proto3`.
2. Add a CMake `add_custom_command` calling `protoc --cpp_out=...` at
configure time, generating `onnx.pb.{h,cc}` into the build directory.
3. Link `Fp16GraphRewriter` against `protobuf::libprotobuf`.
`libprotobuf` is almost certainly already a transitive dep of
`onnxruntime` — confirm by checking
`C:\ANSLibs\onnxruntime\include` for `google/protobuf/...` headers and
`C:\ANSLibs\protobuf` (or vcpkg's protobuf install). Just expose the
include directories to your new target.
**Option 2B: vendor the full ONNX C++ library.**
Bigger dep but provides typed helpers, validators, and a checker. Skip
unless 2A's hand-rolled approach blows up on edge cases.
### Step 3 — Implement `Fp16GraphRewriter` (~2 days for core)
Create `engines/ONNXEngine/Fp16GraphRewriter.h`:
```cpp
#pragma once
#include <string>
#include <vector>
#include <unordered_set>
namespace ANSCENTER {
// Returns the path to an FP16 version of fp32Path. If cacheDir already
// contains a fresh FP16 cache for this model, returns the cached path
// directly (fast path). Otherwise loads fp32Path, runs the in-memory
// FP16 graph rewrite, writes to cacheDir, and returns the new path.
//
// Cache filename embeds a hash of (fp32 file content + opBlockList) so
// a model update or blocklist change automatically invalidates the
// cache — same pattern as TRT engine cache invalidation.
//
// On any failure (parse error, write error, etc.) returns fp32Path
// unchanged and logs a warning via ANS_DBG. The session creation
// continues with the FP32 model — this function NEVER blocks the
// inference path.
std::string ResolveFp16OnnxPath(
const std::string& fp32Path,
const std::string& cacheDir,
const std::vector<std::string>& opBlockList);
} // namespace ANSCENTER
```
Create `engines/ONNXEngine/Fp16GraphRewriter.cpp`. Key components:
#### 3a. IEEE-754 binary32 → binary16 helper
Standard 30-line implementation, round-to-nearest-even, handle NaN/Inf.
Reference: numpy's `numpy/core/src/npymath/halffloat.c` or
`Eigen/src/Core/arch/Default/Half.h`. Self-contained, no deps.
#### 3b. Default op blocklist
```cpp
static const std::unordered_set<std::string> kDefaultFp16BlockList = {
// Object-detection post-processing — known FP16-unsafe
"NonMaxSuppression", "TopK", "RoiAlign",
// Classical ML ops — never have FP16 implementations
"ArrayFeatureExtractor", "Binarizer", "CastMap",
"CategoryMapper", "DictVectorizer", "FeatureVectorizer",
"Imputer", "LabelEncoder", "LinearClassifier",
"LinearRegressor", "Normalizer", "OneHotEncoder",
"RandomUniform", "SVMClassifier", "SVMRegressor",
"Scaler", "TreeEnsembleClassifier", "TreeEnsembleRegressor",
"ZipMap",
};
```
#### 3c. Initializer conversion
Walk `model.graph().initializer()`. For every `TensorProto` with
`data_type == FLOAT`:
1. Pull the float data (it lives in `raw_data` OR `float_data` —
handle both).
2. Convert each float32 to uint16 binary16.
3. Set `data_type = FLOAT16`, clear `float_data`, write the new bytes
to `raw_data`.
Watch for: external data references (when `data_location ==
EXTERNAL`) — for the YOLO and OCR models in this project, weights are
embedded, but write a guard: if external, skip with a warning.
#### 3d. Cast-node insertion at boundaries
For each external input (entries in `model.graph().input()` that are
NOT initializer names):
1. Note the input's tensor name `orig`.
2. Append a new `NodeProto` to the graph: `Cast(input=orig,
output=orig + "_fp16", to=FLOAT16)`.
3. Walk every existing node in the graph; for any input that equals
`orig`, replace it with `orig + "_fp16"`. Be careful not to rewire
the new Cast node itself (skip the last node added).
For each external output: insert `Cast(FP16 → FP32)` symmetrically.
#### 3e. value_info type updates
For every `ValueInfoProto` in `graph().value_info()`:
- If the tensor is the input to OR output of a block-listed op, leave
its `tensor_type.elem_type` as FP32.
- Otherwise update to FP16.
This step is the most error-prone in practice. When in doubt, leave
elem_type alone — the runtime will accept inferred types. Test
correctness empirically with the validation harness in Step 7.
#### 3f. Subgraphs
If/Loop/Scan ops have nested subgraphs (`AttributeProto` of type
GRAPH). Recurse into each. The OCR recognizer doesn't have these;
neither does YOLO post-Ultralytics-export — but write the recursion
defensively, otherwise this fails silently on future models.
#### 3g. Cache management
Cache filename: `<model_basename>.<sha1_8>.fp16.onnx`, where `sha1_8` is
the first 8 hex chars of `SHA1(fp32_file_bytes || serialized
opBlockList || "v1")`. The "v1" string lets you bust the cache for
all models if you fix a converter bug later.
Cache directory: create on demand with `std::filesystem::create_directories`.
Default: `std::filesystem::temp_directory_path() / "ANSCENTER" /
"DMLOptimizedCache"`.
Lookup: if cache file exists AND its mtime is newer than the source
ONNX, use it. Else regenerate.
### Step 4 — Wire into `initialize_handler` (half day)
In `engines/ONNXEngine/ONNXEngine.cpp`, find the DML branch in
`BasicOrtHandler::initialize_handler()`. Insert path resolution before
session creation:
```cpp
case EngineType::AMD_GPU: {
std::string onnxToLoad = _onnx_path;
if (options.dmlFP16) {
const std::string cacheDir =
options.dmlOptimizedModelDir.empty()
? GetDefaultDmlCacheDir()
: options.dmlOptimizedModelDir;
onnxToLoad = ResolveFp16OnnxPath(
_onnx_path, cacheDir, options.dmlFP16OpBlockList);
ANS_DBG("OrtHandler",
"AMD FP16: source=%s cached=%s",
_onnx_path.c_str(), onnxToLoad.c_str());
}
if (!TryAppendDirectML(opts)) { /* existing fallback */ }
ort_session = std::make_unique<Ort::Session>(
env, onnxToLoad.c_str(), opts);
break;
}
```
`GetDefaultDmlCacheDir()` is a small helper in `Fp16GraphRewriter.cpp`.
### Step 5 — Per-model configuration (1 day)
Update the option-builder functions to opt models into FP16:
`modules/ANSOCR/ANSONNXOCR/PaddleOCRV5Engine.cpp` (and equivalents):
```cpp
OrtHandlerOptions BuildLpDetectorOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
o.dmlFP16 = true; // YOLO det validated FP16-safe
return o;
}
OrtHandlerOptions BuildOcrDetectorOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
o.dmlFP16 = true; // DBNet text-det validated
return o;
}
OrtHandlerOptions BuildOcrRecognizerOptions() {
OrtHandlerOptions o;
o.preferTensorRT = true;
o.trtFP16 = true;
// OCR recognizer is FP16-sensitive on small text — keep AMD FP32
// until per-plate accuracy validation passes (see Step 7).
o.dmlFP16 = false;
return o;
}
```
Helmet OD (in `ANSCUSTOMDetector` model loading) — set `dmlFP16 = true`.
### Step 6 — DebugView logging (per `MEMORY.md` rule)
Sprinkle `ANS_DBG` markers at every decision/error point in
`Fp16GraphRewriter.cpp`:
- Entry: `ANS_DBG("Fp16Rewriter", "ResolveFp16OnnxPath: src=%s cache=%s",
fp32Path.c_str(), cacheDir.c_str());`
- Cache hit: `ANS_DBG("Fp16Rewriter", "cache HIT %s", cachedPath.c_str());`
- Cache miss → rewrite: `ANS_DBG("Fp16Rewriter",
"cache MISS — rewriting (initializers=%zu nodes=%zu)", ...);`
- Per phase: log initializer-conversion count, Cast-insertion count,
value_info update count.
- On error: `ANS_DBG("Fp16Rewriter",
"ERROR: %s — falling back to FP32 source", ex.what());` and return
the original FP32 path (never block inference).
### Step 7 — Per-model accuracy validation (1–2 days)
You can't ship FP16 without per-model accuracy validation. Build a
small C++ harness — separate executable target, runs against
labeled fixtures from your test data:
```
tools/AccuracyComparator/AccuracyComparator.cpp
- Loads two sessions (FP32 and FP16) on CPU EP for determinism.
- Feeds N representative frames through both.
- For YOLO det: compare box IoU, score deltas, count of boxes lost.
- For OCR det: compare text-region masks (pixel-wise IoU on the prob
map binary at threshold).
- For OCR rec: compare DECODED CTC strings — not raw logits.
A single character delta on a plate is a regression.
- Emit pass/fail against a configurable tolerance.
```
Acceptance criteria:
| Model | Tolerance | What "pass" means |
|---|---|---|
| YOLO LP detector | mean IoU ≥ 0.95 vs FP32 boxes | No boxes lost, score delta < 0.05 |
| YOLO Helmet OD | mean IoU ≥ 0.95 | Same |
| OCR DBNet | mask IoU ≥ 0.92 at threshold | No new false negatives |
| OCR CTC recognizer | char accuracy within 0.5% | If fails, leave AMD FP32 |
Run on at least 500 frames per model from production data.
### Step 8 — Build, deploy, measure (half day)
Build artifacts: `ANSODEngine.dll`, `ANSOCR.dll`. No new DLL — the
rewriter lives inside the existing engine layer.
Deploy to AMD test box. First run: each session takes an extra
2–10 s during initial cache build. Look for `[Fp16Rewriter] cache
MISS — rewriting` lines.
Subsequent runs: instant cache hit. Look for `[Fp16Rewriter] cache HIT`.
Measure:
- Per-call DML latency in `STATS` lines (mean, p99).
- Total fps over a 60-second run.
- Compare to the always-serial-on-AMD baseline (5.4 fps full-frame ALPR).
- Target: ≥ 7 fps.
If a model regresses accuracy in the field (rare with the validation
harness in Step 7, but possible), flip its `dmlFP16` back to `false`
and ship a hotfix. No code change needed beyond the option flag.
## 6. Dependencies summary
| Dep | Where it lives | New or existing |
|---|---|---|
| `libprotobuf` (C++ runtime) | Almost certainly already vendored via onnxruntime; otherwise add via vcpkg or `C:\ANSLibs\protobuf` | Existing, just expose includes |
| `onnx.proto3` schema file | Vendor at `engines/ONNXEngine/proto/onnx.proto3` from ONNX repo | New, ~1 KB file |
| `protoc` compiler | Comes with libprotobuf install | Existing, just call from CMake |
| `<filesystem>` for cache dir | C++17 standard | Existing |
| SHA1 helper for cache key | Pull from any tiny public-domain impl, or use libcrypto if linked | Tiny new file (~80 lines) |
No third-party FP16 libraries. No Python. No new DLLs in deployment.
## 7. Risks and mitigations
| Risk | Likelihood | Mitigation |
|---|---|---|
| Conversion produces NaN/Inf in some op | Medium | Default blocklist + per-model `dmlFP16OpBlockList` override. Always validate with Step 7 harness before flipping a model on. |
| Some FP16 op has no DML kernel → falls back to CPU silently → SLOWER than FP32 | Medium | Profile each model on AMD with ORT's profiling JSON output (`SessionOptions::EnableProfiling`). If a CPU fallback shows up, blocklist the offending op for that model. |
| ONNX schema version mismatch (we vendor proto3, models use proto with newer fields) | Low | Models from this project are exported by Ultralytics + PaddleOCR — both target stable opset 12-17 ranges that proto3 covers fully. Validate by attempting parse on a known-good model during Step 3. |
| Cache directory permissions on customer machines | Low | Default to `%TEMP%` which is always writable; allow override via `dmlOptimizedModelDir`. On any cache-write failure, fall back to FP32 source path (no inference impact). |
| Subgraph (If/Loop/Scan) bug | Low for current models | Recurse defensively in Step 3f; assert if a subgraph is encountered, fail-safe to FP32. |
## 8. Out of scope (do NOT do as part of this plan)
- **Auto mixed precision** (per-op accuracy-driven precision selection).
That requires running a representative dataset through the model to
decide which ops can safely be FP16. Doable in C++ but adds 1+ week.
Defer to a follow-up plan.
- **Model quantization to INT8.** Different problem, different tooling
(calibration data needed).
- **Switching from DML to MIGraphX or ROCm.** Different EP, different
port. Defer.
- **Recovering depth=2 parallelism on AMD.** The always-serial fix is
in place and stable; this plan is purely about per-call speedup
while remaining serial. Combine the two for net throughput gain.
## 9. Project rules to follow (from MEMORY.md)
- **Edit directly in main checkout.** No worktree, no branch.
- **No commits without explicit user request.** When this plan ships,
don't auto-commit.
- **Sprinkle `ANS_DBG` on new code paths** — entry, decisions, errors.
See Step 6 for exact placement.
## 10. Acceptance checklist
When this plan is done, the following must all be true:
- [ ] `OrtHandlerOptions::dmlFP16` exists and is documented inline.
- [ ] `Fp16GraphRewriter.{h,cpp}` exists with `ResolveFp16OnnxPath` exposed.
- [ ] `BasicOrtHandler::initialize_handler` calls the resolver in the
AMD branch when `dmlFP16` is set.
- [ ] At least one production model (LP YOLO det) is opted in via its
option-builder.
- [ ] Accuracy harness in `tools/AccuracyComparator/` exists and produces
pass/fail for the opted-in models.
- [ ] AMD test-box measurement shows per-call DML latency reduced by
≥ 25% on the opted-in model vs FP32 baseline.
- [ ] No regression on Intel or NVIDIA — the FP16 path never executes
on those EPs (`if (options.dmlFP16)` is gated on `EngineType::AMD_GPU`
via the case-branch placement).
- [ ] DebugView logs cleanly: cache HIT / cache MISS / rewrite phases
/ errors all visible.
- [ ] First-run cache-build latency is ≤ 10 s per model on AMD test box.
- [ ] Subsequent-run cache-load latency is ≤ 100 ms per model.
## 11. Open questions to resolve at start
When you start this plan in a future session, answer these first:
1. Is `libprotobuf` already linked into ANSODEngine.dll (transitively
via onnxruntime)? Run `dumpbin /imports ANSODEngine.dll | grep
protobuf` to confirm. If yes, skip the vcpkg/manual install step.
2. Where does `C:\ANSLibs\onnxruntime\include` live exactly? Confirm
path for the protoc include directive.
3. Which ONNX opset version do your YOLO and OCR models target?
Run `Ort::Session` with profiling once and check the model
metadata. This confirms which `onnx.proto3` revision to vendor.
4. Is the initial AMD measurement still ~5.4 fps with the always-serial
fix in place? If something regressed in the meantime, fix that
first before adding FP16.
---
*Last updated: when this plan was authored. No code shipped yet.*