Files
ANSCORE/docs/PLAN_FFmpeg8_MediaClient_Migration.md

44 KiB
Raw Blame History

PLAN — Migrate MediaClient (and ANSCV) from FFmpeg 4.0.2 to FFmpeg 8.1

Status: Not started Target FFmpeg: n8.1 LGPL shared (C:/ANSLibs/ffmpeg-n8.1) Current FFmpeg: 4.0.2 GPL shared (C:/Projects/CLionProjects/ANSCORE/MediaClient/ffmpeg) Estimated effort: 12 focused days of editing + retesting all audio/video paths Risk: Medium (touches ~10 MediaClient files, audio channel-layout migration is fiddly)


1. Background

ANSCORE's ANSCV.dll currently links against a 2018-era FFmpeg 4.0.2 vendored at MediaClient/ffmpeg/. This is causing two concrete problems:

  1. Hardware encoders don't work. The NVENC wrapper in FFmpeg 4.0 predates the modern NV_ENC_PRESET_CONFIG_VER struct version that current NVIDIA drivers (CUDA 13.x era) expect. avcodec_open2() for hevc_nvenc / h264_nvenc fails with unsupported param (12) regardless of which preset/options we pass.
  2. License entanglement. The current FFmpeg build was compiled with --enable-gpl (it includes libpostproc, libx265, libx264). That makes ANSCV.dll a derivative work of GPL software, which is incompatible with closed-source commercial distribution.

A newer LGPL-only FFmpeg has been downloaded and verified at C:/ANSLibs/ffmpeg-n8.1/. It includes:

  • All hardware encoders we want: hevc_nvenc, hevc_qsv, hevc_amf, h264_*, and av1_nvenc/av1_qsv/av1_amf (AV1 hardware encoding for RTX 40+ / Arc / RX 7000+).
  • Software fallbacks: libsvtav1, libaom-av1, libopenh264, mpeg4.
  • Excluded (LGPL build): libx264, libx265, libxvid, libxavs2, libpostproc, libfdk-aac.
  • License: LGPL v3 (verified in C:/ANSLibs/ffmpeg-n8.1/LICENSE.txt and via the configure string in ffmpeg.exe -version).

Migrating to this build resolves both problems but requires porting MediaClient's C++ code, which uses several FFmpeg APIs that have been removed across the 4.0 → 8.1 jump (5 major versions).

2. Goals and acceptance criteria

The migration is complete when all of the following are true:

  1. ANSCV.dll builds clean with no errors and no new warnings against C:/ANSLibs/ffmpeg-n8.1 headers/libs.
  2. MediaClient/ffmpeg/ is removed from the source tree (no longer referenced).
  3. cmake/Dependencies.cmake points at C:/ANSLibs/ffmpeg-n8.1/.
  4. The new FFmpeg DLLs (avcodec-62.dll, avformat-62.dll, avutil-60.dll, swresample-6.dll, swscale-9.dll, avfilter-11.dll, avdevice-62.dll) are deployed alongside ANSCV.dll runtime location (currently C:/Projects/SharedCore/).
  5. ANSCV_PrintFFmpegLicense_S() prints LGPL version 2.1 or later (or v3) for all four libraries.
  6. ANSCV_ImagesToMP4HW_S() successfully encodes via hevc_nvenc (or av1_nvenc if the GPU is RTX 40+) — the [FF-HW] Using encoder: log line shows a hardware encoder, not a software fallback. NO MORE unsupported param (12) errors.
  7. All existing functional tests pass for: ANSRTSP, ANSRTMP, ANSFLV, ANSMJPEG, ANSSRT, ANSWebcam, ANSVideoPlayer, ANSFilePlayer, ANSFilePlayer_CV, VideoPlayer — i.e. RTSP playback, RTMP playback, FLV file playback, MJPEG streaming, SRT streaming, webcam capture, file player audio/video sync.
  8. No regression in ANSLPR or any other module that consumes ANSCV indirectly.
  9. git status in MediaClient/ shows only the intended changes.

3. Current state (verified facts before starting)

3.1 FFmpeg install

Path Version
Headers C:/Projects/CLionProjects/ANSCORE/MediaClient/ffmpeg/include/ 4.0.2
Import libs C:/Projects/CLionProjects/ANSCORE/MediaClient/ffmpeg/lib/x64/ 4.0.2
Runtime DLLs C:/Projects/SharedCore/avcodec-58.dll, avformat-58.dll, avutil-56.dll, swresample-3.dll, swscale-5.dll, avfilter-8.dll 4.0.2
Build configure --enable-gpl (has libpostproc, libx264, libx265) GPL

Note: C:/Projects/SharedCore/ also already contains avcodec-59.dll, avformat-59.dll, avutil-57.dll from another consumer using FFmpeg 5.x. Leave these in place. The new FFmpeg 8.1 DLLs use yet different major versions (avcodec-62, avformat-62, avutil-60) so all three sets coexist cleanly — Windows DLL loader picks by exact filename.

3.2 New FFmpeg bundle (verified)

C:/ANSLibs/ffmpeg-n8.1/
├── LICENSE.txt        ← LGPL v3, ship this with releases
├── bin/
│   ├── ffmpeg.exe
│   ├── ffplay.exe
│   ├── ffprobe.exe
│   ├── avcodec-62.dll
│   ├── avdevice-62.dll
│   ├── avfilter-11.dll
│   ├── avformat-62.dll
│   ├── avutil-60.dll
│   ├── swresample-6.dll
│   └── swscale-9.dll
├── include/
│   ├── libavcodec/
│   ├── libavdevice/
│   ├── libavfilter/
│   ├── libavformat/
│   ├── libavutil/
│   ├── libswresample/
│   └── libswscale/
└── lib/
    ├── avcodec.lib
    ├── avdevice.lib
    ├── avfilter.lib
    ├── avformat.lib
    ├── avutil.lib
    ├── swresample.lib
    ├── swscale.lib
    └── (.def files, .dll.a files — ignore these, MSVC uses the .lib files)

Encoders confirmed available (via C:/ANSLibs/ffmpeg-n8.1/bin/ffmpeg.exe -encoders):

  • hevc_nvenc, h264_nvenc, av1_nvenc
  • hevc_qsv, h264_qsv, av1_qsv, mjpeg_qsv, mpeg2_qsv
  • hevc_amf, h264_amf, av1_amf
  • libsvtav1, libaom-av1, libvpx-vp9, libopenh264
  • mpeg4 (native, MPEG-4 Part 2)

Encoders NOT available (LGPL build excludes them):

  • libx264, libx265, libxvid, libxavs2

3.3 CMake setup

cmake/Dependencies.cmake lines 150180 define the ffmpeg interface library. As of the start of this plan, line ~171 has already had postproc.lib removed in anticipation of switching to LGPL. The link list currently reads:

if(WIN32)
    target_link_libraries(ffmpeg INTERFACE
        avcodec.lib avdevice.lib avfilter.lib avformat.lib
        avutil.lib swresample.lib swscale.lib
    )
endif()

The include and lib paths still point at the OLD FFmpeg:

set(FFMPEG_INCLUDE_DIR "${CMAKE_SOURCE_DIR}/MediaClient/ffmpeg/include")
set(FFMPEG_LIB_DIR     "${CMAKE_SOURCE_DIR}/MediaClient/ffmpeg/lib/x64")

These need to change. See Phase 2.

3.4 ANSCV functions added in preparation for this migration

These already exist in modules/ANSCV/ANSOpenCV.{h,cpp} and use modern FFmpeg APIs (avcodec_send_frame, avcodec_receive_packet, av_packet_alloc, av_frame_alloc with av_frame_get_buffer, AVChannelLayout-free since they're video-only). They will compile against FFmpeg 8.1 unchanged — but their encoder probe lists hardcode libx265/libx264 which are not in the LGPL build and need updating. See Phase 4.

Class method C export Path
ANSCENTER::ANSOPENCV::ImagesToMP4 ANSCV_ImagesToMP4_S OpenCV VideoWriter (legacy, do NOT touch)
ANSCENTER::ANSOPENCV::ImagesToMP4FF ANSCV_ImagesToMP4FF_S Direct libav*, software-only
ANSCENTER::ANSOPENCV::ImagesToMP4HW ANSCV_ImagesToMP4HW_S Direct libav*, hardware-first
(helper) ANSCV_PrintFFmpegLicense_S Diagnostic — prints library license strings

The ImagesToMP4 (original) function is the user's preserved baseline and must not be modified.

4. Out of scope

  • Updating other ANSLibs dependencies (CUDA, OpenCV, ONNX Runtime, TensorRT, etc.). FFmpeg only.
  • Adding new features beyond what's needed to compile and run.
  • Refactoring MediaClient's overall architecture. This is a port, not a rewrite.
  • Switching audio resampler to swresample if it isn't already (most of MediaClient already uses swresample, but verify).
  • Implementing AV1 hardware encoding configuration in ImagesToMP4HW. That's a follow-up after migration succeeds.
  • Updating cmake-build-release/ cache layout — assume a clean rebuild after the migration.

5. Strategy overview

Five phases, each independently verifiable:

Phase What Verifiable by
1 Preparation: backup, branch, scope review git status clean, backup exists
2 Repoint CMake at new FFmpeg, confirm everything compiles for build targets that don't use MediaClient non-MediaClient targets build clean
3 Port MediaClient C++ files (the hard part) MediaClient compiles, ANSCV.dll links
4 Update ANSCV encoder probe lists for LGPL build NVENC opens, software fallback uses libsvtav1/mpeg4 instead of libx265/libx264
5 Deploy DLLs, run end-to-end tests, validate license All acceptance criteria met

Each phase produces an artifact you can inspect before moving on. Do not skip phases or merge them. The migration risk comes from doing them all at once and not knowing where a failure originated.


Phase 1 — Preparation

1.1 Branch

cd C:/Projects/CLionProjects/ANSCORE
git checkout -b ffmpeg-8-migration
git status   # should be clean except for any uncommitted work

If there's uncommitted work, decide whether to commit it on main first or carry it on the branch.

1.2 Backup the current FFmpeg in case of disaster

cp -r MediaClient/ffmpeg MediaClient/ffmpeg.backup-4.0.2

This is local-only and ignored by git (already in .gitignore patterns or large enough to ignore manually). The backup gives you a one-line revert if everything falls over. Do NOT commit the backup.

1.3 Backup the runtime DLLs

mkdir -p C:/Projects/SharedCore/backup-ffmpeg-4.0.2
cp C:/Projects/SharedCore/avcodec-58.dll \
   C:/Projects/SharedCore/avformat-58.dll \
   C:/Projects/SharedCore/avutil-56.dll \
   C:/Projects/SharedCore/swresample-3.dll \
   C:/Projects/SharedCore/swscale-5.dll \
   C:/Projects/SharedCore/avfilter-8.dll \
   C:/Projects/SharedCore/backup-ffmpeg-4.0.2/

Do NOT delete the live -58.dll files yet. Other binaries deployed to SharedCore may still depend on them (and the -59.dll set is for yet another consumer). All three sets need to coexist after the migration.

1.4 Phase 1 acceptance

  • git status shows only the new branch.
  • MediaClient/ffmpeg.backup-4.0.2/ exists and has the same size as MediaClient/ffmpeg/.
  • C:/Projects/SharedCore/backup-ffmpeg-4.0.2/ exists and contains 6 DLLs.

Phase 2 — Repoint CMake and confirm baseline

2.1 Edit cmake/Dependencies.cmake

Find the FFmpeg block (around lines 162180). Change the include and lib paths from MediaClient/ffmpeg/... to the absolute path C:/ANSLibs/ffmpeg-n8.1/....

Before:

else()
    set(FFMPEG_INCLUDE_DIR "${CMAKE_SOURCE_DIR}/MediaClient/ffmpeg/include")
    set(FFMPEG_LIB_DIR     "${CMAKE_SOURCE_DIR}/MediaClient/ffmpeg/lib/x64")
    add_library(ffmpeg INTERFACE)
    target_include_directories(ffmpeg INTERFACE ${FFMPEG_INCLUDE_DIR})
    target_link_directories(ffmpeg INTERFACE ${FFMPEG_LIB_DIR})
    if(WIN32)
        target_link_libraries(ffmpeg INTERFACE
            avcodec.lib avdevice.lib avfilter.lib avformat.lib
            avutil.lib swresample.lib swscale.lib
        )
    else()
        target_link_libraries(ffmpeg INTERFACE
            avcodec avdevice avfilter avformat
            avutil swresample swscale
        )
    endif()
    message(STATUS "FFmpeg: using ANSLibs at ${FFMPEG_INCLUDE_DIR}")
endif()

After:

else()
    # FFmpeg 8.1 LGPL shared build
    # See docs/PLAN_FFmpeg8_MediaClient_Migration.md
    if(WIN32)
        set(FFMPEG_INCLUDE_DIR "C:/ANSLibs/ffmpeg-n8.1/include")
        set(FFMPEG_LIB_DIR     "C:/ANSLibs/ffmpeg-n8.1/lib")
    else()
        set(FFMPEG_INCLUDE_DIR "${ANSLIBS_DIR}/ffmpeg-n8.1/include")
        set(FFMPEG_LIB_DIR     "${ANSLIBS_DIR}/ffmpeg-n8.1/lib")
    endif()
    add_library(ffmpeg INTERFACE)
    target_include_directories(ffmpeg INTERFACE ${FFMPEG_INCLUDE_DIR})
    target_link_directories(ffmpeg INTERFACE ${FFMPEG_LIB_DIR})
    if(WIN32)
        target_link_libraries(ffmpeg INTERFACE
            avcodec.lib avdevice.lib avfilter.lib avformat.lib
            avutil.lib swresample.lib swscale.lib
        )
    else()
        target_link_libraries(ffmpeg INTERFACE
            avcodec avdevice avfilter avformat
            avutil swresample swscale
        )
    endif()
    message(STATUS "FFmpeg: using ANSLibs at ${FFMPEG_INCLUDE_DIR}")
endif()

Note: postproc.lib was already removed during preparation and stays removed (LGPL builds don't have it).

2.2 Wipe CMake cache and reconfigure

rm -rf cmake-build-release/CMakeCache.txt cmake-build-release/CMakeFiles
# Then trigger CMake reconfigure, either through CLion or:
cmake -S . -B cmake-build-release -G Ninja -DCMAKE_BUILD_TYPE=Release

The reconfigure should succeed. The configure-time message(STATUS "FFmpeg: ...") should print C:/ANSLibs/ffmpeg-n8.1/include.

2.3 Phase 2 acceptance

  • cmake-build-release/CMakeCache.txt shows FFMPEG_INCLUDE_DIR=C:/ANSLibs/ffmpeg-n8.1/include.
  • An attempted build will fail (because MediaClient hasn't been ported yet) but the failures should all be inside MediaClient files, not anywhere else. If a non-MediaClient file breaks, investigate before continuing.

Expected error pattern at this point:

MediaClient/media/audio_decoder.cpp(116): error C2039: 'av_init_packet': is not a member of '...'
MediaClient/media/avcodec_mutex.cpp(41): error C3861: 'avcodec_close': identifier not found
MediaClient/media/audio_encoder.cpp(115): error C2039: 'channel_layout': is not a member of 'AVCodecContext'
... etc

Save this error output. It's the punch list for Phase 3.


Phase 3 — Port MediaClient C++ code

This is the work-heavy phase. It's mechanical but there are ~20 call sites across ~6 files. Do them in the order below — earlier files unblock later ones.

3.1 Files to touch (complete list)

# File What needs to change
1 MediaClient/media/avcodec_mutex.cpp Remove avcodec_close() call
2 MediaClient/media/avcodec_mutex.h Possibly drop avcodec_close() declaration if exposed
3 MediaClient/media/audio_decoder.cpp av_init_packet, channel_layout, av_get_default_channel_layout
4 MediaClient/media/audio_decoder.h Update if it stores channel info as int channels + uint64_t channel_layout
5 MediaClient/media/audio_encoder.cpp channel_layout, av_get_default_channel_layout (4 sites)
6 MediaClient/media/audio_encoder.h Update if it stores channel info
7 MediaClient/media/video_decoder.cpp Remove avcodec_close() call
8 MediaClient/media/video_decoder.h Possibly minor — check after .cpp is fixed
9 MediaClient/media/video_player.cpp av_init_packet (3 sites: lines 274, 458, 829)
10 MediaClient/media/file_player.cpp av_init_packet (2 sites: lines 427, 749)

There may be additional fallout from the channel-layout migration that surfaces only after the obvious cases are fixed. Plan to do two compile-fix iterations: first to address the API removals, second to address any cascading errors (e.g. struct field access that no longer compiles because the field type changed).

3.2 API change reference

These five removals are the entire scope for MediaClient. Examples below.

3.2.1 av_init_packet(&pkt)av_packet_alloc() / av_packet_free()

av_init_packet() was removed in FFmpeg 5.0. In modern FFmpeg, AVPacket is opaque — you allocate it with av_packet_alloc() and free it with av_packet_free(). There is no longer a stack-allocatable AVPacket struct.

Before:

AVPacket packet;
av_init_packet(&packet);
packet.data = some_buffer;
packet.size = some_size;

int ret = avcodec_send_packet(ctx, &packet);
// ... use packet ...

// (no explicit free needed for stack packet in old code)

After:

AVPacket* packet = av_packet_alloc();
if (!packet) {
    // handle allocation failure
    return -1;
}
packet->data = some_buffer;
packet->size = some_size;

int ret = avcodec_send_packet(ctx, packet);
// ... use packet ...

av_packet_free(&packet);  // sets packet to nullptr after freeing

Per-call-site checklist for each av_init_packet:

  • Find the corresponding scope where the packet's lifetime ends (function return, end of block, error exit path).
  • Allocate with av_packet_alloc() at the same place av_init_packet used to be called.
  • Add av_packet_free(&pkt) at every exit path (return statement, end of scope, error branches). RAII via std::unique_ptr<AVPacket, decltype(&av_packet_free_wrapper)> is cleaner if there are many exit paths — define a one-line wrapper because av_packet_free takes AVPacket** not AVPacket*.
  • Replace all &packet/packet.field with packet/packet->field.

Specific sites:

File Line (4.0.2) Context
audio_decoder.cpp 116 Inside Decode() or similar, packet fed from input buffer
file_player.cpp 427 Inside read loop
file_player.cpp 749 Inside read/seek path
video_player.cpp 274 Inside decode-while-streaming
video_player.cpp 458 Inside another decode path
video_player.cpp 829 Inside encode/relay path

Line numbers are from the unmodified 4.0.2-era file. After the first fix in a file, subsequent line numbers shift — re-find by name.

3.2.2 avcodec_close(ctx) → just remove it

avcodec_close() was removed in FFmpeg 7.0. In modern FFmpeg, avcodec_free_context() does both close and free in one call. Existing code that does both avcodec_close() then avcodec_free_context() should drop the close.

Before:

if (m_pContext) {
    avcodec_close(m_pContext);
    avcodec_free_context(&m_pContext);
}

After:

if (m_pContext) {
    avcodec_free_context(&m_pContext);
}

Specific sites:

  • MediaClient/media/avcodec_mutex.cpp:41
  • MediaClient/media/video_decoder.cpp:286

In avcodec_mutex.cpp, the function may be a wrapper that exists specifically to take a global lock around avcodec_close() (some older FFmpeg APIs needed mutex protection). If so, decide whether the wrapper is still needed at all — modern avcodec_free_context() is internally thread-safe per-context. Read the file before deciding. It may just need the body to call avcodec_free_context instead, OR the entire wrapper may be dead code now.

3.2.3 AVCodecContext::channel_layout (uint64_t) and AVCodecContext::channelsAVCodecContext::ch_layout (AVChannelLayout)

This is the most invasive change. The old API:

AVCodecContext* ctx;
ctx->channels = 2;
ctx->channel_layout = AV_CH_LAYOUT_STEREO;  // uint64_t bitmask

The new API (FFmpeg 5.1+, mandatory in 7.0+):

AVCodecContext* ctx;
av_channel_layout_default(&ctx->ch_layout, 2);
// OR explicitly:
AVChannelLayout layout = AV_CHANNEL_LAYOUT_STEREO;
av_channel_layout_copy(&ctx->ch_layout, &layout);

AVChannelLayout is a struct with these fields you typically read:

  • int nb_channels — replaces the old channels int
  • enum AVChannelOrder order — usually AV_CHANNEL_ORDER_NATIVE
  • uint64_t mask (when order is NATIVE) — equivalent to the old channel_layout uint64_t

To get the channel count from a stream/codec context:

// Old:
int n = ctx->channels;

// New:
int n = ctx->ch_layout.nb_channels;

Functions that take channel layouts changed signatures. Notable ones used in MediaClient:

  • av_get_default_channel_layout(int n) -> uint64_tremoved. Use:

    AVChannelLayout layout;
    av_channel_layout_default(&layout, n);
    // ... use layout ...
    av_channel_layout_uninit(&layout);
    
  • swr_alloc_set_opts(SwrContext*, uint64_t out_ch_layout, AVSampleFormat out_fmt, int out_rate, uint64_t in_ch_layout, AVSampleFormat in_fmt, int in_rate, int log_offset, void* log_ctx) -> SwrContext*removed. Use:

    SwrContext* swr = nullptr;
    AVChannelLayout out_layout, in_layout;
    av_channel_layout_default(&out_layout, out_channels);
    av_channel_layout_default(&in_layout,  in_channels);
    int ret = swr_alloc_set_opts2(
        &swr,
        &out_layout, out_fmt, out_rate,
        &in_layout,  in_fmt,  in_rate,
        0, nullptr);
    // (don't forget to swr_init() afterwards)
    av_channel_layout_uninit(&out_layout);
    av_channel_layout_uninit(&in_layout);
    

    Note swr_alloc_set_opts2 (with the 2) — different signature, takes AVChannelLayout* instead of uint64_t, and now allocates+sets in one call writing into &swr.

Specific sites:

File Line (4.0.2) Field/Function
audio_encoder.cpp 115 m_pCodecCtx->channel_layout = av_get_default_channel_layout(m_EncoderParams.DstChannels);
audio_encoder.cpp 280 m_pFrame->channel_layout = av_get_default_channel_layout(m_EncoderParams.DstChannels);
audio_encoder.cpp 375 m_pFrame->channel_layout = av_get_default_channel_layout(m_EncoderParams.SrcChannels);
audio_encoder.cpp 403 m_pResampleFrame->channel_layout = av_get_default_channel_layout(m_EncoderParams.DstChannels);
audio_decoder.cpp 65 m_pContext->channel_layout = av_get_default_channel_layout(channels);
audio_decoder.cpp 223 m_pResampleFrame->channel_layout = av_get_default_channel_layout(m_nChannels);

For AVCodecContext writes: replace with av_channel_layout_default(&m_pCodecCtx->ch_layout, n);. The nb_channels field is set automatically by the helper.

For AVFrame writes: same pattern — av_channel_layout_default(&m_pFrame->ch_layout, n);. AVFrame also got a new ch_layout field in 5.1+. The old AVFrame::channel_layout and AVFrame::channels fields are deprecated then removed.

Be careful in audio_encoder.cpp lines 280, 375, 403 — these write to AVFrame->channel_layout, not AVCodecContext. Same migration pattern but on the frame's ch_layout field.

Audit pass needed: after fixing these six obvious sites, search all of MediaClient/media/ for any other reference to:

  • ->channels where the left side is an AVCodecContext* or AVFrame* (not the RTSP channels array — be careful)
  • ->channel_layout similarly
  • av_get_default_channel_layout
  • swr_alloc_set_opts( (without the 2)
  • AV_CH_LAYOUT_* macros — usually still work but may need to be wrapped in AVChannelLayout initialization

A safer grep is: grep -nE 'channel_layout|->channels[^_]|av_get_default_channel_layout|swr_alloc_set_opts\(' MediaClient/media/*.cpp MediaClient/media/*.h

3.2.4 m_pContext->codec->capabilities — likely still works, verify

AVCodecContext::codec (the pointer to the const AVCodec* the context was opened with) is still present in FFmpeg 8.1. The capability check m_pContext->codec->capabilities & AV_CODEC_CAP_DELAY should compile unchanged.

If it doesn't (some niche removal I'm not remembering), the modern alternative is:

const AVCodec* codec = avcodec_find_decoder(ctx->codec_id);  // or _encoder
if (codec && (codec->capabilities & AV_CODEC_CAP_DELAY)) { ... }

Sites to verify:

  • audio_encoder.cpp:311
  • audio_decoder.cpp:166
  • video_decoder.cpp:964

These are sanity checks during the cleanup loop — if the encoder supports DELAY mode, drain residual frames. The check itself doesn't need changes unless the header organization changed (it didn't, in 8.1).

3.2.5 Other things to watch for

These weren't surfaced by the initial grep but are common 4.0 → 8.1 breakage points. Search for them after the obvious fixes:

Removed Replacement
AVStream::codec (the embedded AVCodecContext) Allocate your own AVCodecContext, fill from AVStream::codecpar via avcodec_parameters_to_context
AVFormatContext::filename AVFormatContext::url
AVFrame::pkt_pts AVFrame::pts (use directly)
AVFrame::pkt_duration AVFrame::duration
av_frame_get_pkt_duration(frame) frame->duration
av_frame_get_pkt_pts(frame) frame->pts
av_register_all(), avcodec_register_all(), avformat_network_init() (for av_register_all) Just delete the call
AVCodec* (non-const pointers) const AVCodec* — required since 5.0
pkt.convergence_duration Removed; check if used and remove

The above are common; not all may apply to MediaClient. Don't proactively edit anything not flagged by the compiler — fix what fails, leave the rest.

3.3 Suggested per-file workflow

For each file, in order:

  1. Open the file.
  2. Re-grep within the file to find all current call sites of removed APIs. The line numbers in this plan are from the 4.0.2-era unmodified file and shift after the first edit.
  3. Apply changes in order from bottom of file to top (so earlier line numbers stay valid for later edits in the same pass).
  4. Compile just this file with the new headers if your build system allows isolated compilation; otherwise, do a project build and look for errors in this file specifically.
  5. Iterate until the file compiles clean.
  6. Move to the next file.

Suggested order (dependency-light first, dependency-heavy last):

  1. avcodec_mutex.cpp — smallest, isolated
  2. video_decoder.cpp — uses avcodec_close, no audio churn
  3. video_player.cpp — three av_init_packet, video only
  4. file_player.cpp — two av_init_packet, video focus
  5. audio_decoder.cpp — channel layout migration
  6. audio_encoder.cpp — channel layout migration (the most extensive)

After all six compile cleanly, do a full project build and look for cascading failures elsewhere.

3.4 Phase 3 acceptance

  • cmake --build cmake-build-release --target ANSCV --config Release succeeds with zero errors.
  • Warnings are acceptable but should be reviewed — -Wdeprecated-declarations warnings on FFmpeg API are the only category that's fine to ignore (they're informational about future removals).
  • ANSCV.dll is produced in cmake-build-release/lib/ (or wherever the project's output dir is).
  • dumpbin /dependents cmake-build-release/lib/ANSCV.dll (or cmake-build-release/bin/) shows imports for avcodec-62.dll, avformat-62.dll, avutil-60.dll, swresample-6.dll, swscale-9.dll, avfilter-11.dll, avdevice-62.dll — NOT the -58/-56/-3/-5/-8 versions.

Phase 4 — Update ANSCV encoder probe lists for LGPL build

The ImagesToMP4FF and ImagesToMP4HW functions in modules/ANSCV/ANSOpenCV.cpp currently include libx265 and libx264 in their probe lists. These encoders don't exist in the LGPL build and will be skipped at runtime. That's not a bug, but the lists will be cleaner and the fallback path will use better encoders if updated.

4.1 New encoder probe order (replaces existing list in ImagesToMP4HW)

1. av1_nvenc       — NVIDIA AV1   (RTX 40+)
2. av1_qsv         — Intel AV1    (Arc, 13th-gen iGPU+)
3. av1_amf         — AMD AV1      (RX 7000+)
4. hevc_nvenc      — NVIDIA HEVC  (Maxwell+, all modern NVIDIA)
5. hevc_qsv        — Intel HEVC   (Skylake+)
6. hevc_amf        — AMD HEVC     (most modern AMD)
7. h264_nvenc      — NVIDIA H.264 (universal NVIDIA fallback)
8. h264_qsv        — Intel H.264
9. h264_amf        — AMD H.264
10. libsvtav1      — software AV1, multithreaded, fast in 8.1
11. libopenh264    — software H.264 (BSD-licensed — fully LGPL-clean fallback)
12. mpeg4          — last resort, LGPL native FFmpeg encoder

Removed from list (no longer present in LGPL build):

  • libx265
  • libx264

4.2 Encoder option dictionaries to use (modern NVENC preset names work in 8.1)

After the FFmpeg update, NVENC's p1p7 preset names are valid (they were added in FFmpeg 4.4). Replace the legacy slow preset names with the new ones — the new ones offer a more granular quality/speed trade-off and unlock options like tune=hq which the older names didn't support.

Recommended option dicts after the update:

Encoder Options
av1_nvenc {rc=vbr, cq=30, preset=p5, tune=hq, multipass=fullres, spatial_aq=1, temporal_aq=1, rc-lookahead=32}
hevc_nvenc {rc=vbr, cq=28, preset=p5, tune=hq, multipass=fullres, spatial_aq=1, temporal_aq=1, rc-lookahead=32, b_ref_mode=middle}
h264_nvenc {rc=vbr, cq=24, preset=p5, tune=hq, multipass=fullres, spatial_aq=1, temporal_aq=1, rc-lookahead=32, b_ref_mode=middle, weighted_pred=1}
av1_qsv {global_quality=30, preset=slower, look_ahead=1, look_ahead_depth=40}
hevc_qsv {global_quality=28, preset=slower, look_ahead=1, look_ahead_depth=40}
h264_qsv {global_quality=24, preset=slower, look_ahead=1, look_ahead_depth=40}
av1_amf {quality=quality, rc=cqp, qp_i=30, qp_p=32, qp_b=34}
hevc_amf {quality=quality, rc=cqp, qp_i=24, qp_p=26, qp_b=28}
h264_amf {quality=quality, rc=cqp, qp_i=22, qp_p=24, qp_b=26}
libsvtav1 {crf=32, preset=6} (preset 0=slowest/best, 13=fastest, 6 is balanced)
libopenh264 {} (limited tuning available; defaults are decent)
mpeg4 (no opts, set codec_ctx->bit_rate = 1500000)

Pixel formats per encoder:

  • NVENC: AV_PIX_FMT_YUV420P (works), or AV_PIX_FMT_NV12 (slightly faster, no conversion needed inside the encoder)
  • QSV: AV_PIX_FMT_NV12 (required)
  • AMF: AV_PIX_FMT_NV12 (preferred)
  • libsvtav1, libopenh264, mpeg4: AV_PIX_FMT_YUV420P

4.3 What to update in ImagesToMP4FF

ImagesToMP4FF is the software-only variant. After the LGPL switch, replace its current list:

libx265 → libx264 → mpeg4   (current)

with:

libsvtav1 → libopenh264 → mpeg4

Or even simpler, since libsvtav1 is fast and produces small files, and libopenh264 covers H.264, the fallback chain is already sensible.

4.4 Phase 4 acceptance

  • ImagesToMP4HW and ImagesToMP4FF no longer reference libx265 or libx264 in their encoder lists.
  • The build still succeeds (this is a pure constant/data change).
  • Functional behavior verified in Phase 5.

Phase 5 — Build, deploy, test, validate

5.1 Deploy the new FFmpeg DLLs to SharedCore

cp C:/ANSLibs/ffmpeg-n8.1/bin/avcodec-62.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/avformat-62.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/avutil-60.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/swresample-6.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/swscale-9.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/avfilter-11.dll \
   C:/ANSLibs/ffmpeg-n8.1/bin/avdevice-62.dll \
   C:/Projects/SharedCore/

The old -58, -56, -3, -5, -8 DLLs from FFmpeg 4.0 stay in place. The -59, -57, -4, -6, -7 DLLs from FFmpeg 5.x (used by another consumer) also stay in place. After this step, SharedCore contains DLLs from THREE FFmpeg generations side by side, each used by whichever binary was linked against it.

5.2 Full clean rebuild

rm -rf cmake-build-release/CMakeCache.txt cmake-build-release/CMakeFiles
cmake -S . -B cmake-build-release -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build cmake-build-release --config Release -j

Expect the build to take longer than usual (clean rebuild). All targets should succeed.

5.3 Smoke test 1: license verification

Run the unit test (or any binary that calls ANSCV_PrintFFmpegLicense_S()):

./cmake-build-release/bin/ANSCV-UnitTest.exe

Expected output:

[FFmpeg] avutil   license: LGPL version 2.1 or later   (or v3)
[FFmpeg] avcodec  license: LGPL version 2.1 or later
[FFmpeg] avformat license: LGPL version 2.1 or later
[FFmpeg] swscale  license: LGPL version 2.1 or later

If it shows GPL — STOP. Either the wrong build was deployed or CMake is still pointing at the old FFmpeg.

5.4 Smoke test 2: hardware encoder probe

Run ImagesToMP4HW against a known input folder:

ANSCV_ImagesToMP4HW_S("E:/Programs/DemoAssets/ImageSeries", "E:/output_hw_test.mp4", 0, 10);

Expected log:

[FF-HW Thread ...] Image: 1920x1085 -> Video: 1920x1084 | 26 frames @ 10 FPS (~2s)
[FF-HW] Using encoder: NVIDIA HEVC (NVENC) ... (or similar - depends on GPU)
[FF-HW] Video created: ...\output_hw_test.mp4 (26 frames, 10 FPS, ~2s) via NVIDIA HEVC (NVENC)

The log should NOT contain:

  • unsupported param (12) (FFmpeg version mismatch — means the wrong DLL is loaded)
  • Cannot get the preset configuration (same)
  • Function not implemented (same)
  • A fall-through to libsvtav1 (means hardware is actually unavailable on this machine — NOT a migration failure)

5.5 Smoke test 3: MediaClient functionality

This is the riskiest part — verifying that the MediaClient port didn't break anything. Run each of the following manually with known-good inputs:

Subsystem Test
ANSRTSP Connect to a known RTSP camera, verify video frames flow and audio (if expected) plays
ANSRTMP Connect to a known RTMP source, verify video frames flow
ANSFLV Open a known FLV file, verify video and audio play in sync
ANSMJPEG Connect to a known MJPEG HTTP stream, verify frames
ANSSRT Connect to a known SRT stream, verify frames
ANSWebcam Open a USB webcam, verify capture
ANSVideoPlayer Open a known H.264 video file, verify HW decode (NVDEC/D3D11VA/DXVA2 path) works
ANSFilePlayer / ANSFilePlayer_CV Open a known video file, verify SW decode path works
VideoPlayer Same as above, alternate API

Audio paths are the highest-risk because of the channel layout migration. Listen for:

  • Silence where there should be audio
  • Distorted/sped-up/slowed-down audio (sample rate or channel count mismatch)
  • Channel swap (left and right channels reversed)
  • Crashes in swr_* resampler functions

If any of these fail, the audio channel layout migration almost certainly missed a site or has a wrong channel count somewhere.

5.6 Phase 5 acceptance

All boxes ticked:

  • Clean rebuild succeeds with zero errors
  • ANSCV_PrintFFmpegLicense_S reports LGPL
  • ImagesToMP4HW opens NVENC successfully (assuming the test machine has any modern NVIDIA GPU)
  • All 9 MediaClient subsystems verified manually (table in 5.5)
  • No regression in any module that uses ANSCV (ANSLPR, etc.)

If all boxes check, the migration is complete. Commit and merge.


6. Cleanup (optional, after verification)

Once the migration is verified stable on main and not just the branch, you can:

  1. Delete MediaClient/ffmpeg/ (the old vendored FFmpeg source). It's no longer referenced by anything.

  2. Delete the backup directories (MediaClient/ffmpeg.backup-4.0.2/ and C:/Projects/SharedCore/backup-ffmpeg-4.0.2/) once you're confident in the new build.

  3. Optionally remove the old DLLs from SharedCore (avcodec-58.dll, avformat-58.dll, avutil-56.dll, swresample-3.dll, swscale-5.dll, avfilter-8.dll) — but only if no other binary in the field still depends on them. Check with whoever owns the broader product deployment first. Removing these DLLs when other consumers still rely on them WILL break those consumers at runtime with a DLL not found error.

  4. Add a THIRDPARTY.txt to your release artifacts containing the LGPL v3 notice and a link to FFmpeg's source (to satisfy LGPL §6 obligations). Include the contents of C:/ANSLibs/ffmpeg-n8.1/LICENSE.txt in your distribution.

7. Rollback plan

If anything goes wrong and you need to revert:

# 1. Restore the old FFmpeg headers/libs (if you went through with deletion)
cp -r MediaClient/ffmpeg.backup-4.0.2 MediaClient/ffmpeg

# 2. Revert the CMake change
git checkout main -- cmake/Dependencies.cmake

# 3. Wipe the build cache and rebuild
rm -rf cmake-build-release/CMakeCache.txt cmake-build-release/CMakeFiles
cmake -S . -B cmake-build-release -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build cmake-build-release --config Release -j

# 4. Discard the migration branch (or keep it for later)
git checkout main
git branch -D ffmpeg-8-migration   # only if abandoning entirely

The new FFmpeg DLLs in SharedCore (avcodec-62.dll etc.) can stay — they don't conflict with anything. Only ANSCV.dll's link table determines which DLL it loads, and a rolled-back ANSCV.dll will be linked against the old -58.dll set again.

8. Risk register

Risk Likelihood Impact Mitigation
Audio channel layout migration introduces silent or distorted audio High Medium Manual testing of every audio path in 5.5, side-by-side comparison with old build
A 4.0 → 8.1 API removal that I haven't catalogued surfaces during compile Medium Low Mechanical fix per occurrence; FFmpeg's release notes (linked below) document every removal
Some other consumer of C:/Projects/SharedCore/ was relying on avcodec-58.dll and we delete it during cleanup Low High Only delete during cleanup phase 6, and only after verifying no other live binaries depend on the old DLLs
The LGPL build's lack of libx265 means ImagesToMP4FF software fallback degrades quality Low Low Phase 4 swaps in libsvtav1 + libopenh264 which produce equivalent or better quality
NVENC still fails after the migration (e.g. NVIDIA driver still incompatible) Very Low Medium FFmpeg 8.1 supports current and recent NVIDIA driver generations; if it fails, update NVIDIA driver as a separate step
MediaClient/media/*.h headers expose more FFmpeg types than expected, causing cascading edits Medium Medium Plan deliberately tackles .cpp files first; header changes follow only if compilation requires

9. Reference materials

Appendix A — Complete file modification checklist

Print this and tick boxes as you go.

MediaClient C++ source

  • MediaClient/media/avcodec_mutex.cpp — remove avcodec_close() (line 41, may be in a wrapper function)
  • MediaClient/media/avcodec_mutex.h — drop wrapper declaration if entire wrapper is now redundant
  • MediaClient/media/audio_decoder.cppav_init_packet (line 116), channel_layout (lines 65, 223), av_get_default_channel_layout (lines 65, 223)
  • MediaClient/media/audio_decoder.h — verify channel-related public fields/methods still match new API
  • MediaClient/media/audio_encoder.cppchannel_layout (lines 115, 280, 375, 403), av_get_default_channel_layout (same lines)
  • MediaClient/media/audio_encoder.h — verify channel-related public fields/methods
  • MediaClient/media/video_decoder.cppavcodec_close() (line 286)
  • MediaClient/media/video_decoder.h — verify no breakage
  • MediaClient/media/video_player.cppav_init_packet (lines 274, 458, 829)
  • MediaClient/media/file_player.cppav_init_packet (lines 427, 749)

ANSCV C++ source (for Phase 4)

  • modules/ANSCV/ANSOpenCV.cppImagesToMP4FF encoder list: replace libx265/libx264 with libsvtav1/libopenh264/mpeg4
  • modules/ANSCV/ANSOpenCV.cppImagesToMP4HW encoder list: add av1_nvenc/av1_qsv/av1_amf at the top, replace libx265/libx264 with libsvtav1/libopenh264/mpeg4 in the software fallback section
  • modules/ANSCV/ANSOpenCV.cppImagesToMP4HW NVENC option dicts: switch from legacy preset names to p5 + tune=hq + spatial/temporal AQ + multipass

CMake

  • cmake/Dependencies.cmakepostproc.lib already removed (Phase 0 / preparation)
  • cmake/Dependencies.cmake — repoint FFMPEG_INCLUDE_DIR and FFMPEG_LIB_DIR to C:/ANSLibs/ffmpeg-n8.1

Deployment

  • Copy 7 new FFmpeg DLLs to C:/Projects/SharedCore/
  • Add THIRDPARTY.txt with LGPL notice to release artifacts
  • Include LICENSE.txt from FFmpeg bundle in release artifacts

Testing

  • License print is LGPL
  • ImagesToMP4HW uses NVENC (or appropriate hardware)
  • RTSP playback works (video and audio)
  • RTMP playback works
  • FLV playback works
  • MJPEG streaming works
  • SRT streaming works
  • Webcam capture works
  • HW video decoder (ANSVideoPlayer) works
  • SW video decoder (ANSFilePlayer, VideoPlayer) works
  • ANSLPR and other downstream modules unaffected

Appendix B — Quick API reference

Old (FFmpeg 4.0) New (FFmpeg 8.1) Removed in
av_init_packet(&pkt) then declare AVPacket pkt; AVPacket* pkt = av_packet_alloc(); ... av_packet_free(&pkt); 5.0
avcodec_close(ctx) (delete) — avcodec_free_context(&ctx) covers it 7.0
ctx->channels = N; ctx->channel_layout = X; av_channel_layout_default(&ctx->ch_layout, N); 7.0
frame->channels, frame->channel_layout frame->ch_layout.nb_channels, frame->ch_layout 7.0
av_get_default_channel_layout(N) (returns uint64_t) av_channel_layout_default(&layout, N) (writes to AVChannelLayout) 7.0
swr_alloc_set_opts(swr, out_layout_u64, ...) swr_alloc_set_opts2(&swr, &out_layout, ...) 7.0
AVCodec* (non-const) const AVCodec* 5.0
AVStream::codec (embedded AVCodecContext) Allocate own AVCodecContext, fill from AVStream::codecpar 5.0
AVFormatContext::filename AVFormatContext::url 5.0
AVFrame::pkt_pts AVFrame::pts 6.0
AVFrame::pkt_duration AVFrame::duration 7.0
av_register_all() (delete) — automatic 5.0
avcodec_register_all() (delete) — automatic 5.0
pkt.convergence_duration (removed, no replacement, just delete) 6.0

Appendix C — Known-good CMake snippet

After Phase 2, cmake/Dependencies.cmake lines 162180 should look like this (Windows path version):

else()
    # FFmpeg 8.1 LGPL shared build (BtbN distribution)
    # Migration plan: docs/PLAN_FFmpeg8_MediaClient_Migration.md
    if(WIN32)
        set(FFMPEG_INCLUDE_DIR "C:/ANSLibs/ffmpeg-n8.1/include")
        set(FFMPEG_LIB_DIR     "C:/ANSLibs/ffmpeg-n8.1/lib")
    else()
        set(FFMPEG_INCLUDE_DIR "${ANSLIBS_DIR}/ffmpeg-n8.1/include")
        set(FFMPEG_LIB_DIR     "${ANSLIBS_DIR}/ffmpeg-n8.1/lib")
    endif()
    add_library(ffmpeg INTERFACE)
    target_include_directories(ffmpeg INTERFACE ${FFMPEG_INCLUDE_DIR})
    target_link_directories(ffmpeg INTERFACE ${FFMPEG_LIB_DIR})
    if(WIN32)
        target_link_libraries(ffmpeg INTERFACE
            avcodec.lib avdevice.lib avfilter.lib avformat.lib
            avutil.lib swresample.lib swscale.lib
        )
    else()
        target_link_libraries(ffmpeg INTERFACE
            avcodec avdevice avfilter avformat
            avutil swresample swscale
        )
    endif()
    message(STATUS "FFmpeg: using ANSLibs at ${FFMPEG_INCLUDE_DIR}")
endif()

postproc.lib is intentionally absent — the LGPL build does not include libpostproc.

Appendix D — Quick test commands

# Verify FFmpeg version and configuration after deployment
C:/ANSLibs/ffmpeg-n8.1/bin/ffmpeg.exe -version

# List available encoders
C:/ANSLibs/ffmpeg-n8.1/bin/ffmpeg.exe -hide_banner -encoders | grep -iE "nvenc|qsv|amf|svtav1|openh264|mpeg4"

# Smoke-test NVENC HEVC encoding directly (independent of ANSCV)
C:/ANSLibs/ffmpeg-n8.1/bin/ffmpeg.exe \
    -y -hide_banner \
    -framerate 10 \
    -pattern_type glob -i "E:/Programs/DemoAssets/ImageSeries/*.jpg" \
    -c:v hevc_nvenc -preset p5 -tune hq -rc vbr -cq 28 \
    -movflags +faststart \
    E:/test_nvenc_direct.mp4

# Confirm ANSCV.dll's import table after rebuild
dumpbin /dependents cmake-build-release/lib/ANSCV.dll | grep -i "av\|sw\|postproc"
# Should show avcodec-62.dll, avformat-62.dll, avutil-60.dll, swresample-6.dll, swscale-9.dll
# Should NOT show avcodec-58.dll, postproc-*.dll

End of plan. Last updated when initial draft was written. Update as the migration progresses with notes about anything discovered along the way.