Drop avcodec_close for FFmpeg 8 compatibility

[baldvin-kovacs]

Problem

On systems with FFmpeg 8.0+ (Arch as of late 2025/early 2026, recent
Fedora, etc.), KasmVNC cannot initialize hardware video encoding. The
log shows:

ffmpeg: Failed to load symbol avcodec_close
VNCServerST: Hardware video encoding acceleration capability: unavailable

KasmVNC then silently falls back to its software WebP/JPEG encoders,
which is a major regression for users who configured videoCodec: h264_vaapi (or similar) and are now paying significant CPU for
something the hardware was meant to handle.

Root cause

avcodec_close() was deprecated in FFmpeg 3.x (its replacement,
avcodec_free_context(), has existed since 2015) and was finally
removed in FFmpeg 8.0.

FFmpeg::FFmpeg() resolves every libavcodec symbol via the
D_LOOKUP_SYM macro, which throws on any missing symbol. The
constructor catches the throw and sets available = false —
disabling the entire hardware-encode path. So the missing
avcodec_close symbol kills hardware encoding wholesale, even though
the encode path itself never calls avcodec_close.

Why removing the symbol is safe

grep -rn 'avcodec_close' over the tree shows only one production
caller: the destructor of FfmpegFrameFeeder (a benchmarking helper
for Xvnc_replay_benchmark):

FfmpegFrameFeeder::~FfmpegFrameFeeder() {
    ffmpeg->avcodec_close(codec_ctx_guard.get());
}

That call was already redundant. codec_ctx_guard is a
FFmpeg::ContextGuard (a std::unique_ptr whose deleter calls
avcodec_free_context). When the destructor returns, the
unique_ptr is destroyed and avcodec_free_context runs — and
avcodec_free_context internally already closes the codec before
freeing the AVCodecContext. The explicit avcodec_close was a
no-op in terms of correctness, and on FFmpeg 8 it's just impossible.

The production VNC encode path uses the same guard pattern and
therefore was already working correctly via avcodec_free_context;
it never needed the avcodec_close shim.

Change

- Drop the avcodec_close_func typedef, member, dlsym, and wrapper
method from common/rfb/ffmpeg.{h,cxx}.
- Default the benchmark destructor (= default) — the unique_ptr
guard handles cleanup, which is the modern idiomatic approach.

Diff is 1 insertion, 10 deletions across 3 files. No behavior change
on FFmpeg ≤ 7; restores hardware encode on FFmpeg ≥ 8.

Verification

Built against FFmpeg 8.1.1 on Arch (libavcodec 62.x).

Before:
ffmpeg: Failed to load symbol avcodec_close
VNCServerST: Hardware video encoding acceleration capability: unavailable

After:
ffmpeg: libavutil.so loaded
ffmpeg: libswscale.so loaded
ffmpeg: libavcodec.so loaded
VNCServerST: Hardware video encoding acceleration capability: h264_vaapi

intel_gpu_top during a streamed-video session confirms the iGPU's
VCS (Video) engine wakes from RC6 and stays active throughout the
session, where previously it was a flat 0% across every sample.

[ttimonen]

LGTM