# NextNote Windows FFmpeg Build Plan

## Purpose

Create a reproducible, auditable Windows x64 FFmpeg build containing only the capabilities NextNote uses. The output will replace the current third-party Gyan `ffmpeg.exe` in NextNote Display, NextNote PowerPoint Helper, and NextNote Google Slides Helper.

This plan is an implementation handoff. It does not authorize changing application behaviour beyond the FFmpeg packaging, extraction, and verified encoder support described here.

## Current state

- All three Windows applications currently embed the same `ffmpeg.7z` archive.
- The archive contains one `ffmpeg.exe` built as `FFmpeg 8.1 essentials_build-www.gyan.dev`.
- The legacy embedded executable includes `h264_nvenc`, `h264_amf`, `h264_qsv`, and `libx264`.
- The PowerPoint and Google Slides helpers select supported hardware encoders in this order:
  1. NVIDIA NVENC
  2. AMD AMF
  3. Intel Quick Sync/QSV
- AMD must precede QSV because an AMD system can also expose an Intel iGPU with Quick Sync.
- Display and PowerPoint Helper extract to `%AppData%\Visionmill\NextNote\ffmpeg.exe`.
- Google Slides Helper extracts to `%AppData%\Visionmill\NextNote GS Helper\ffmpeg.exe`.
- The existing extractors return an already-present file without checking its version or hash. This would prevent an updated FFmpeg binary from reaching an existing installation and must be fixed before rollout.

Current embedded archive SHA-256 at the time of planning:

```text
f2f9e6d5173d282016fe941818bee321d5953d9553cdbd4962a98d75bf2ee1a5
```

## Goals

1. Produce a native, static Windows x64 `ffmpeg.exe` from pinned and verified sources.
2. Build only the FFmpeg command-line executable; do not ship `ffprobe`, `ffplay`, development headers, or unrelated tools.
3. Support every command currently issued by the three NextNote applications.
4. Support NVIDIA, AMD, and Intel hardware H.264 encoding.
5. Support NextNote Display's raw H.264 decoding to BGRA raw video through
   vendor-neutral Windows D3D11VA hardware decode.
6. Make the build reproducible from a clean Windows machine using one documented entry command.
7. Produce the binary, licence notices, exact corresponding sources, build configuration, dependency lock, checksums, and test report as one release set.
8. Replace stale extracted binaries safely when the embedded FFmpeg version changes.
9. Make the result suitable for later integration into Windows Release Manager.

## Non-goals

- Do not replace FFmpeg with an in-process media library.
- Do not change the fixed NextNote ports or network protocol.
- Do not add audio capture or encoding.
- Do not add AV1, HEVC, VP8, VP9, media containers, playback UI, or transcoding features that NextNote does not use.
- Do not add `--enable-nonfree`; a non-free build cannot be redistributed.
- Do not silently fall back to software encoding or decoding. Slides and notes
  remain available when LiveView hardware requirements are not met.
- Do not attempt a macOS FFmpeg build in this project.

## Repository and deliverable structure

Implement inside this folder:

```text
Windows/FFmpeg Build/
├── PROJECT_PLAN.md
├── README.md
├── build.ps1
├── test.ps1
├── config/
│   ├── dependencies.lock.json
│   ├── ffmpeg.configure.args
│   └── required-capabilities.json
├── scripts/
│   ├── bootstrap-msys2.ps1
│   ├── fetch-and-verify.ps1
│   ├── build-dependencies.sh
│   ├── build-ffmpeg.sh
│   ├── verify-binary.ps1
│   ├── package-release.ps1
│   └── update-nextnote-assets.ps1
├── tests/
│   ├── capability-tests.ps1
│   ├── encode-smoke-tests.ps1
│   └── decode-smoke-tests.ps1
└── artifacts/
    └── <version>-<build-id>/
```

Generated source trees, MSYS2 packages, and build intermediates must live under a local ignored `work/` folder. Do not commit generated toolchains or binaries to the build-project source tree. The release artifact folder is retained as a historical build record.

No source file should become a monolithic release script. `build.ps1` orchestrates the focused scripts above and contains no build implementation duplicated from them.

## Supported application command surface

The final binary must support the following exact functional surface.

### Helper capture and encode

- Windows `gdigrab` input.
- Desktop capture with `desktop`, `framerate`, `video_size`, `offset_x`, `offset_y`, and `draw_mouse` options.
- `lavfi` input and the `color` source for the pre-encoded black frame.
- Video filters: `scale`, `format`, and `drawbox`.
- Encoders:
  - `h264_nvenc`
  - `h264_amf`
  - `h264_qsv`
- Raw Annex-B H.264 output through `pipe:1`.
- H.264 bitstream filters:
  - `dump_extra`
  - `filter_units`
- Null muxer for encoder capability probes.
- High profile, level 4.0, `yuv420p`, no B-frames, short GOP, low-latency operation, and the encoder-specific options used by the helpers.

### Display decode

- Raw H.264 input from `pipe:0`.
- D3D11VA H.264 hardware decoding on a compatible Intel, NVIDIA, or AMD GPU.
- No production software-decoding fallback: LiveView requires compatible
  D3D11VA hardware and drivers.
- `rawvideo` output to `pipe:1`.
- BGRA pixel output and swscale conversion.
- Runtime options used by the display, including `probesize`, `analyzeduration`, `nobuffer`, `flush_packets`, `low_delay`, and passthrough frame timing.

### Output formats

Only the formats needed by NextNote are required:

- raw H.264 demuxer and muxer
- rawvideo muxer
- null muxer
- pipe/file protocols needed by the command line

## Technical approach

### Build host

Use a native Windows x64 build host with a PowerShell entry point and a pinned MSYS2 UCRT64/MinGW toolchain.

Reasons:

- It produces a native Windows executable on the same operating system used for application release testing.
- It avoids making WSL or Docker a release prerequisite.
- It is scriptable from Windows Release Manager later.
- MSYS2 provides the Unix build environment expected by FFmpeg while PowerShell owns validation, signing, packaging, and application-asset updates.

The scripts must not depend on Visual Studio being open. If a Windows SDK signing tool is used, discover it using the same robust discovery approach as Windows Release Manager rather than a fixed SDK version path.

### Version and source pinning

Start with FFmpeg 8.1.2, the current stable 8.1 release when this plan was created. Never download an unpinned branch during a release build.

`config/dependencies.lock.json` must contain, for every source input:

- component name
- exact version, tag, or commit SHA
- canonical source URL
- archive filename
- SHA-256
- upstream licence
- reason it is included

Pin these inputs:

- FFmpeg 8.1.2 release source and detached PGP signature
- AMD AMF headers exact release/tag or commit SHA
- NVIDIA `nv-codec-headers` exact release/tag
- Intel oneVPL/libvpl exact release/tag
- MSYS2 installer/package snapshot or an explicitly recorded package set and versions
- 7-Zip version used for final packaging

The first green build may establish the compatible dependency pins. Once established, release builds must use the lock unchanged until an intentional dependency-update review.

### Source verification

`fetch-and-verify.ps1` must:

1. Download only the URLs in the dependency lock.
2. Verify every SHA-256 before extraction.
3. Verify the FFmpeg release PGP signature with the official FFmpeg release key.
4. Fail closed on any mismatch, missing signature, unexpected redirect, or missing source.
5. Record the verification result in the build manifest.

Do not silently substitute a newer release when a source is unavailable.

### Configure strategy

Use a two-stage implementation:

1. Build a reproducible compatibility binary with all required NextNote capabilities.
2. Minimise the configuration only after the full automated and hardware test matrix passes.

The final configure file should begin from a disabled baseline and explicitly enable the required programs, libraries, protocols, devices, filters, demuxers, muxers, decoders, encoders, parsers, and bitstream filters. Claude must derive the exact dependency closure from FFmpeg 8.1.2's `configure`; it must not guess option names.

Mandatory policy:

```text
--enable-amf
--enable-nvenc
--enable-ffnvcodec
--enable-libvpl
```

`--enable-gpl`, `--enable-version3`, `--enable-libx264`, and
`--enable-nonfree` are prohibited. The standalone executable must report
FFmpeg's LGPL licence and contain no GPL encoder. Build statically so the
application continues to deploy one `ffmpeg.exe`. Publish the matching FFmpeg
source, linked dependency sources, build configuration, scripts, notices, and
LGPL text with every release.

The resulting `ffmpeg -buildconf` output is part of the release artifact and must exactly match the checked-in configure arguments after normalised path differences.

### Reproducibility

The build must:

- run from a clean checkout using `build.ps1`
- avoid timestamps or local absolute paths in the binary where the toolchain permits
- use a stable build root
- set deterministic locale and timezone for build metadata
- record the compiler, linker, assembler, shell, MSYS2 packages, and dependency versions
- generate SHA-256 hashes for every release file
- run a second clean build and compare hashes before claiming byte-for-byte reproducibility

If byte-identical output is not initially possible, document the exact differing regions and still require identical capability lists, source locks, configure arguments, and test results. Do not label the build reproducible until the byte comparison passes.

### Signing

After all FFmpeg tests pass, Authenticode-sign `ffmpeg.exe` with the existing Visionmill Azure Trusted Signing workflow. Signing is an integrity and publisher step; it does not replace or alter FFmpeg's LGPL notices.

Record both unsigned and signed SHA-256 values in the manifest. Package only the signed executable in application assets.

### Application asset update

`update-nextnote-assets.ps1` must accept a completed, verified build artifact and update these three embedded archives with the identical signed `ffmpeg.exe`:

```text
Windows/NextNote/NextNoteDisplay/Assets/ffmpeg.7z
Windows/NextNote/NextNoteHelper/Assets/ffmpeg.7z
Windows/NextNote/NextNote Google Slides/NextNoteGSHelper/Assets/ffmpeg.7z
```

Before writing it must:

- verify the input build is marked passed
- verify the signed binary hash against the manifest
- create deterministic archives with the same internal filename, `ffmpeg.exe`
- verify all three generated archives have the same SHA-256
- verify each archive contains exactly one file
- refuse to proceed if any target path is missing

It must not build or publish the applications.

### Extraction/version migration

Update all three application extractors before shipping the custom build.

Required design:

1. Embed a small FFmpeg manifest alongside `ffmpeg.7z` containing at least build ID, executable SHA-256, and archive SHA-256.
2. On startup, hash the canonical extracted `ffmpeg.exe`.
3. If the hash matches the embedded manifest, reuse it.
4. If it differs, stop or confirm no app-owned FFmpeg process is active, extract to a temporary file, verify the temporary hash, and atomically replace the canonical file.
5. Never leave a partially written executable after interruption.
6. Preserve an explicit, separately configured developer override path if the product genuinely supports one. Do not treat an arbitrary stale file in the canonical app-data path as a user override.
7. Log the old and new build IDs without logging personal paths unnecessarily.

This migration must be tested by installing over a release that has already extracted the current Gyan binary.

## Hardware encoder policy

The application, tests, and documentation must use this order:

```text
h264_nvenc → h264_amf → h264_qsv
```

Rationale:

- Prefer discrete GPU hardware encoding.
- AMD must precede QSV so an Intel iGPU does not mask a supported AMD GPU.
- If no encoder initialises, keep slides and notes working and show a clear
  LiveView hardware-requirement message.

Required AMD low-latency settings currently used by both helpers:

```text
-usage ultralowlatency
-quality speed
-rc vbr_peak
-b:v 2M
-maxrate 4M
-bufsize 1M
-profile:v high
-level 4.0
-pix_fmt yuv420p
```

The AMF probe and live command must both succeed on the supported AMD test machine. Do not infer AMF support merely because `ffmpeg -encoders` lists `h264_amf`; the probe must actually initialise and encode frames with the installed driver.

## Hardware decoder policy

The Windows Display explicitly requests FFmpeg's D3D11VA hardware acceleration.
The custom build must expose D3D11VA H.264 decoding. D3D11VA is preferred over
vendor-specific decoder selection because it uses the installed Windows driver
for Intel, NVIDIA, and AMD hardware. LiveView is unavailable when no compatible
D3D11VA device or driver is available; the Display does not silently degrade to
software decoding.

Quick Sync encoder detection is not a valid test for Display hardware decoding.
An NVIDIA-only system can fail the QSV encoder probe while still providing
fully supported H.264 hardware decode through D3D11VA.

## Automated verification

`test.ps1` must fail the build if any required check fails.

### Static capability checks

Capture and inspect:

```powershell
ffmpeg.exe -version
ffmpeg.exe -buildconf
ffmpeg.exe -devices
ffmpeg.exe -formats
ffmpeg.exe -filters
ffmpeg.exe -encoders
ffmpeg.exe -decoders
ffmpeg.exe -hwaccels
ffmpeg.exe -bsfs
ffmpeg.exe -protocols
```

Assert all capabilities named in this plan are present. Also assert known-unneeded programs/codecs/formats are absent according to `required-capabilities.json` where absence is deliberate.

### Hardware encode/decode smoke test

Using the selected hardware encoder on an interactive Windows desktop:

1. Capture a small desktop region (QSV-compatible input) or generate a black
   source for NVENC/AMF.
2. Encode it with the first successfully initialised hardware encoder as raw
   Annex-B H.264.
3. Decode it from stdin to BGRA raw video on stdout.
4. Assert a zero exit code and non-empty expected-size output.
5. Exercise `dump_extra` and `filter_units` in the encode command.

### Capture smoke test

On an interactive Windows desktop:

1. Capture a small `gdigrab` desktop region.
2. Apply the actual NextNote scale/filter path.
3. Encode two or more frames to the null muxer with each available encoder.
4. Encode a raw H.264 sample through a pipe.

The test must distinguish:

- encoder not compiled
- supported but no matching hardware/driver on this test machine
- compiled and successfully initialised
- command/option incompatibility

### Hardware matrix

Run and retain test reports for:

| Machine | Expected selected encoder | Required result |
|---|---|---|
| NVIDIA GPU | `h264_nvenc` | LiveView stable; no fallback |
| AMD GPU, including Intel/AMD hybrid if available | `h264_amf` | AMF selected before QSV |
| Intel QSV without NVIDIA/AMD | `h264_qsv` | LiveView stable |
| no supported hardware encoder | none | LiveView unavailable; slides and notes remain functional |

Each hardware test must cover:

- encoder probe and cache update
- starting/stopping LiveView repeatedly
- slide transitions
- black-frame generation when presentation ends
- reconnecting a Display during the stream
- 10, 20, and 30 fps settings
- both PowerPoint Helper and Google Slides Helper
- decoding on NextNote Display
- absence of FFmpeg process leaks after quit

## Packaging and compliance deliverables

Every completed build artifact must include:

```text
ffmpeg.exe
ffmpeg.7z
build-manifest.json
SHA256SUMS.txt
ffmpeg-version.txt
ffmpeg-buildconf.txt
ffmpeg-capabilities.txt
test-report.json
THIRD_PARTY_NOTICES.txt
COPYING.LGPLv2.1
SOURCE_README.txt
nextnote-ffmpeg-8.1.2-sources.7z
sources/
```

`sources/` and `nextnote-ffmpeg-8.1.2-sources.7z` must contain the exact
corresponding source for FFmpeg and all statically linked dependencies, local
patches, build scripts, dependency lock, configure arguments, and enough
instructions to rebuild the binary. A link to upstream alone is not sufficient
for the release record.

Publish the corresponding-source bundle on the same NextNote web server as the application downloads. The existing `/source/ffmpeg/8.1/` URL may remain the stable public entry point, but it must clearly identify the exact 8.1.2 build and provide its matching sources, checksums, licence text, configure line, and build instructions.

When the custom binary ships, update application credits and website wording from “Gyan FFmpeg build” to “NextNote FFmpeg 8.1.2 build” and retain attribution to the FFmpeg project and all linked components. Review the EULA and download pages against FFmpeg's compliance guidance. This plan is technical guidance, not legal advice; obtain qualified legal review for commercial H.264 patent and LGPL obligations.

## Milestones

### Milestone 0 — Baseline inventory

Deliver:

- current archive and executable hashes
- current Gyan `-version`, `-buildconf`, and capability output
- machine-readable inventory of every FFmpeg command and option used by all three applications
- baseline CPU and available-hardware test results

Exit criteria:

- `required-capabilities.json` accounts for every current invocation.
- No build work begins with an unrecorded application requirement.

### Milestone 1 — Pinned build environment

Deliver:

- dependency lock
- source download and verification scripts
- MSYS2 bootstrap and package inventory
- clean environment validation

Exit criteria:

- a clean Windows x64 machine can fetch and verify every input without manual file copying.
- tampering with any input causes a hard failure.

### Milestone 2 — Compatibility build

Deliver:

- signed Windows x64 `ffmpeg.exe`
- complete build manifest and logs
- all three hardware H.264 encoders compiled
- required capture, filter, pipe, raw H.264, rawvideo, and decode capabilities

Exit criteria:

- all static capability and hardware encode/decode smoke tests pass.
- binary runs on a clean supported Windows installation without MSYS2 installed.

### Milestone 3 — Hardware qualification

Deliver:

- NVIDIA, AMD, and Intel reports
- confirmed selection order `NVENC → AMF → QSV`
- real PowerPoint and Google Slides LiveView tests

Exit criteria:

- AMD is selected on an Intel/AMD hybrid test machine.
- no regression is observed on previously supported NVIDIA or Intel hardware.

### Milestone 4 — Minimalisation

Deliver:

- reduced configure surface
- before/after capability diff and binary size comparison
- repeated complete test suite

Exit criteria:

- no required capability is removed.
- all Milestone 2 and 3 tests still pass.
- every remaining enabled external component is documented and licensed.

### Milestone 5 — Application integration

Deliver:

- identical updated archives in all three application assets
- embedded FFmpeg manifest
- hash-aware, atomic extraction in all three applications
- upgrade-over-existing-install tests

Exit criteria:

- an existing extracted Gyan binary is replaced automatically by the new verified build.
- a matching extracted binary is not rewritten.
- a corrupt extraction is rejected and safely recovered.

### Milestone 6 — Release and source publication

Deliver:

- final signed binary and archive
- exact corresponding-source bundle
- public web source page/files
- updated credits/EULA/download attribution
- completed release record

Exit criteria:

- public binary and source hashes match the release manifest.
- application installers contain the same FFmpeg archive hash.
- download, install, update, LiveView, and uninstall tests pass.

### Milestone 7 — Windows Release Manager integration

**Complete.** A focused FFmpeg profile invokes the established scripts rather
than reimplementing build logic. It can prepare a PGP-verified latest release
as a separate candidate lock, then starts with an unsigned candidate build.
Explicit production signing and asset-publish selections are required before
it stages a separate executable copy, packages with the passed hardware
qualification report, and invokes the asset-update script.

Exit criteria:

- Release Manager can start a clean build, show verification/test/signing stages, retain failed and completed history, and reveal the artifact.
- a failed FFmpeg build cannot update application assets.

## Final acceptance criteria

The work is complete only when all of the following are true:

- A clean Windows x64 machine can produce the release by following `README.md` and running one top-level build command.
- All sources and toolchain inputs are pinned and verified.
- `ffmpeg.exe -version` identifies the intended FFmpeg 8.1.2 source and NextNote build metadata without claiming to be an unmodified upstream binary.
- The executable exposes `h264_nvenc`, `h264_amf`, and `h264_qsv`, and does
  not expose `libx264`.
- PowerPoint Helper and Google Slides Helper select NVENC, then AMF, then QSV.
- AMF genuinely initialises and streams on AMD hardware.
- Display decodes the resulting raw H.264 stream from every encoder.
- Black-frame generation works for AMF as well as the other encoders.
- Existing installations replace their old extracted FFmpeg binary safely.
- All three embedded archives are byte-identical.
- No `--enable-nonfree` component or configuration is present.
- Exact corresponding source and build materials are published and linked from the application download area.
- The complete Windows release test matrix passes without relying on MSYS2 or developer tools on the end-user machine.

## Risks and controls

| Risk | Control |
|---|---|
| AMF is listed but fails with an installed driver | Real-frame startup probe and AMD hardware test; automatic QSV fallback |
| QSV masks AMD on hybrid machines | AMF is explicitly ordered before QSV in probe and fallback chains |
| Over-minimisation breaks a rarely used command | Generate requirements from source invocations; compatibility build before minimisation |
| Dependency update breaks FFmpeg configure compatibility | Pin the first proven compatible tags/commits; update intentionally and rerun full matrix |
| Existing app retains stale FFmpeg | Embedded manifest plus hash-aware atomic replacement |
| LGPL/source compliance is incomplete | Package exact source, patches, scripts, lock, licences, and configure line with every release |
| Hardware encoder output differs in NAL layout | Test packet/keyframe handling and reconnect behaviour for every encoder |
| Build is only reproducible on one developer machine | Clean-machine bootstrap test and recorded tool/package versions |
| Signed and tested binaries differ | Test first, sign, hash again, then run post-sign smoke tests and package only the signed hash |

## Research ledger

### Chosen

- FFmpeg 8.1.2 stable source, verified with the FFmpeg release PGP key.
- Native Windows x64 build using PowerShell plus pinned MSYS2 UCRT64/MinGW tooling.
- Static single-executable distribution to match the existing application architecture.
- LGPL-2.1-or-later standalone build with all GPL and non-free components disabled.
- NVENC, AMF, then QSV selection order; no software encoder fallback.
- Build a compatibility binary first, then minimise with regression tests.
- D3D11VA as the required Windows vendor-neutral H.264 hardware-decode path,
  requested explicitly rather than through FFmpeg automatic selection.
- For the v1.0.1 external-FFmpeg path, retain one persistent Avalonia
  `WriteableBitmap` per LiveView session and update it in place on the UI thread.
  Do not allocate a new native bitmap for every decoded frame.
- Treat codec qualification and end-to-end display cadence as separate release
  gates. See `WINDOWS_LIVEVIEW_ARCHITECTURE_REVIEW.md`.

### Rejected

- Continuing indefinitely with an opaque third-party binary: easy initially, but does not give NextNote a controlled, locked build and corresponding-source release process.
- WSL or Docker as a mandatory release dependency: adds a second operating environment and complicates integration with the Windows release/signing workflow.
- Visual Studio-only FFmpeg build: FFmpeg and most linked dependencies are better supported through their configure/MinGW build paths.
- `--enable-nonfree`: output would not be redistributable.
- GPL/libx264 build: rejected after explicit product approval on 2026-07-24;
  it creates unnecessary GPL obligations and permits a poor software fallback
  experience on unsupported hardware.
- Putting QSV before AMF: an Intel iGPU could prevent AMD hardware encoding from being selected.

### Decision pending

- Replacing Windows Display's external FFmpeg decoder with the native Media
  Foundation H.264 decoder and D3D11/Avalonia composition interop. The intended
  path is an `ID3D11Texture2D` decode surface exposed through a shared NT handle
  and imported into an Avalonia `CompositionDrawingSurface`. This is the closest
  Windows equivalent to the Swift VideoToolbox/CVPixelBuffer/Metal path, but it
  requires an isolated proof before production migration.
- Releasing the custom FFmpeg build in v1.0.1. Hardware qualification is
  necessary but is not sufficient; `ffplay` isolation and complete Display
  cadence testing must also pass.

### Primary sources

- FFmpeg downloads, releases, and signature verification: https://ffmpeg.org/download.html
- FFmpeg legal and licence guidance: https://ffmpeg.org/legal.html
- FFmpeg device documentation, including `gdigrab`: https://ffmpeg.org/ffmpeg-devices.html
- FFmpeg configure source: https://github.com/FFmpeg/FFmpeg/blob/master/configure
- FFmpeg command-line hardware acceleration and D3D11VA documentation:
  https://ffmpeg.org/ffmpeg.html
- Microsoft Media Foundation H.264 decoder and DXVA awareness:
  https://learn.microsoft.com/windows/win32/medfound/h-264-video-decoder
- Avalonia custom rendering and external GPU image import:
  https://docs.avaloniaui.net/docs/graphics-animation/custom-rendering
- AMD AMF FFmpeg integration: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/FFmpeg-and-AMF-HW-Acceleration
- AMD AMF encoder settings: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/wiki/AMF-Encoder-Settings-and-Tuning-in-FFmpeg
- Intel oneVPL: https://github.com/intel/libvpl
- NVIDIA codec headers: https://github.com/FFmpeg/nv-codec-headers/releases

## Handoff instruction

Implement this plan milestone by milestone and stop at each exit criterion for review. Do not update the three application assets until the standalone binary has passed the compatibility and available-hardware tests. Windows-only builds and hardware tests must be run by the user on Windows; do not claim them as passed from macOS.
