FrameFlow.Inference.Dml 0.9.0-alpha.1

FrameFlow

Cross-platform FFmpeg-based media playback for .NET, with a UI-agnostic core.

FrameFlow decodes and plays audio and video on Windows, Linux and macOS. The playback core has no UI dependency: A/V sync, seeking, and buffering happen behind interfaces, and presenters attach at the edges. Avalonia and SDL presenters ship in the box, and you can write your own.

Pre-1.0. Public surface and internal contracts change freely between releases. There are no external consumers yet, so the bias is toward getting the shape right rather than staying compatible. Each change is listed with its fix in docs/BREAKING-CHANGES.md.

Install

Packages are on nuget.org. You need the library plus the native FFmpeg binaries, which ship as a separate runtime package:

dotnet add package FrameFlow.Player --prerelease

dotnet add package FrameFlow.Native.Runtime --prerelease

Requires .NET 10. Add a presenter and an audio backend for the platform you target — FrameFlow.Avalonia, FrameFlow.Sdl, FrameFlow.Audio.OpenAL. The full list is under Packages.

Quick start

Start from FrameFlowPlayer.Open. One builder, two terminals — pick by what you need from playback:

BuildPlayerAsync — the full player

using FrameFlow.Audio.OpenAL;
using FrameFlow.Media;
using FrameFlow.Player;

await using var player = await FrameFlowPlayer.Open(path)
    .WithOpenAlAudio()          // also implements IClockSource, so it becomes the master clock
    .WithAvaloniaVideoView(view)
    .WithHardwareDecode(HardwareDecodeMode.Auto)
    .WithRepeatMode(RepeatMode.All)
    .BuildPlayerAsync();

var played = await player.PlayAsync();
if (!played.IsSuccess)
    Console.Error.WriteLine($"{played.Error.Category}: {played.Error.Message}");

MediaPlayer.CreateAsync(...) is the positional form of the same thing — both it and the builder run the same wiring. Reach for it directly when you already hold all eleven arguments; most of the examples in this repository still do. WithClock is the one option it cannot express, so a chain that injects a clock has to end at BuildPlayerAsync.

BuildAsync — play to end of stream

When you only need "open a file and play it to the end", with no seek, pause, or repeat:

await using var player = await FrameFlowPlayer.Open(path)
    .WithAudioSink(audioSink)   // .WithAvaloniaVideoView(view) / .WithOpenAlAudio() also available
    .BuildAsync();

await player.PlayToCompletionAsync(ct);

WithRepeatMode, WithClock, WithHardwareFrames and WithAudioActivation mean nothing to a PlayerSession. Calling any of them narrows the chain to IMediaPlayerBuilder, where BuildPlayerAsync is the only terminal on offer, so the mismatch is a compile error rather than an ignored setting.

Generic Host and DI

services.AddFrameFlow() registers the engine's environment pieces: the OpenAL backend, the FFmpeg bootstrap as a hosted service, the Avalonia video sink, and options. The playback session itself stays an explicitly created runtime object — resolve the registered sinks and hand them to one of the surfaces above rather than resolving a player singleton:

builder.Services
    .AddFrameFlow()
    .AddFrameFlowOpenAlAudio()   // registers IAudioSink (container-owned)
    .AddHostedBootstrap();       // FFmpeg bootstrap runs at host startup

// …then, inside an IHostedService, resolve IAudioSink and build the session:
await using var player = await FrameFlowPlayer.Open(path)
    .WithAudioSink(resolvedAudioSink)
    .BuildAsync(ct);

Errors

Transport commands on IPlaybackController and IMediaPlayer return Result rather than throwing. A command the state machine refuses — a seek on a non-seekable source, a play on a disposed player — is an expected outcome, and Result.Error carries an ErrorCategory alongside the message:

var seeked = await player.SeekAsync(TimeSpan.FromSeconds(30));
if (!seeked.IsSuccess && seeked.Error.Category == ErrorCategory.InvalidOperation)
    DisableTheSeekBar();

IsSuccess carries [MemberNotNullWhen(false, nameof(Error))], so a failure branch reads Error without a null check.

Exceptions still mean what exceptions mean. MediaPlayer.CreateAsync throws if it cannot build a player, argument validation throws, and anything a sink or the decode stack raises comes through. Failures that arise mid-playback rather than in answer to a command surface on IMediaPlayer.ErrorOccurred.

See ADR-0069.

What works

• software decode and a hardware-decode path
• a zero-copy Windows presenter that hands GPU frames straight to a D3D composition-interop surface
• OpenAL audio output on all three platforms, doubling as the master clock
• Avalonia and SDL presenters
• camera capture and an H.264 to MP4 encoder
• optional DirectML and CUDA inference: YOLO detection, Whisper captioning

11 runnable example apps under examples/ exercise these against real files and live camera and multicast sources.

Packages

FrameFlow.Native.Runtime carries the FFmpeg binaries. The libraries do not reference it — add it yourself, or supply the natives another way.

Any package here works on its own. The FFmpeg resolver installs itself on the first native call, so FrameFlow.Decoding opens a file without a bootstrap call and without a dependency on the player layer (ADR-0070). Bootstrap explicitly — AddHostedBootstrap(), or new FrameFlowBootstrapper(options).Initialize() — when you need to choose which binaries load, or need the hardware-decode capabilities the result reports. Do it before the first decode call: libraries load once per process.

FrameFlow.MotionClip is a camera-tracked motion-clip capture tool. It is not on nuget.org; take the self-contained binary from Releases.

Building from source

FrameFlow needs FFmpeg shared libraries on disk. They are gitignored, so prime them once per clone:

dotnet run scripts/fetch-ffmpeg.cs

That writes into runtimes/{rid}/native/, which Directory.Build.targets copies into every project's output. Then:

dotnet build ./FrameFlow.slnx --nologo

The whole solution restores from nuget.org alone. Six projects take a PackageReference on FrameFlow.Native.Runtime for self-contained publish — FrameFlow.MotionClip and the AvaloniaPlayer, Camera.Inference.Dml, DualPlayer, Multicast.Dml and ZeroCopyInterop examples. That package is mapped to nuget.org by exact id in nuget.config; the FrameFlow.* prefix is deliberately not mapped there, so a new FrameFlow PackageReference needs its id added.

scripts/fetch-cuda.cs (CUDA execution provider) and scripts/generate-test-corpus.cs (integration-test media) are documented in scripts/README.md.

Tests

19 test projects live under tests/. The integration suite needs the FFmpeg runtimes and a generated corpus:

dotnet run scripts/generate-test-corpus.cs

dotnet test ./FrameFlow.slnx --nologo

scripts/run-tests.sh is faster — it fans one dotnet test process out per project, and needs a prior dotnet build.

A handful of tests open a real SDL window and are skipped unless FRAMEFLOW_VISUAL_TESTS=1. Nothing sets it, including CI, so presenter and windowing regressions are not caught by normal validation. Run them deliberately, on a machine with a display:

FRAMEFLOW_VISUAL_TESTS=1 dotnet test ./tests/FrameFlow.Integration.Tests --nologo

tests/frameflow.runsettings pins the gate to 0 and injects it into the test host, so passing -settings tests/frameflow.runsettings overrides an ambient FRAMEFLOW_VISUAL_TESTS=1. Use one or the other.

Documentation

• docs/BREAKING-CHANGES.md — what moved in each release and what to write instead
• docs/ARCHITECTURE.md — the layering and the reasoning behind it
• docs/adr/ — architectural decision records, the authority on what was decided and why
• CONTRIBUTING.md — build prerequisites, test corpus, commit convention

The other directories under docs/ are project history. ROADMAP.md and phases/ record how the project got here, investigations/ holds dated bug and perf write-ups, and archive/ holds superseded material.

Contributing

Not accepting contributions. Pull requests will not be reviewed or merged.

Bug reports are welcome in the issue tracker, with no promise of a reply.

Security problems go through the private advisory form, not the issue tracker — see SECURITY.md.

License

FrameFlow is released under the PolyForm Small Business License 1.0.0.

It is source-available, not open source: the license is not OSI-approved, though it does carry the SPDX identifier PolyForm-Small-Business-1.0.0. In short, you may use, modify and distribute FrameFlow for any purpose provided your company has fewer than 100 people and less than USD 1,000,000 (2019, inflation-adjusted) in prior-year revenue. Personal, noncommercial, educational and evaluation use are permitted regardless of company size. LICENSE.md is the authority; this paragraph is not.

If your company is over those thresholds, contact the maintainer about a commercial license.

Third-party components

FrameFlow's own license does not extend to the components it builds on. The significant ones:

The pinned FFmpeg build is a prebuilt LGPL archive from BtbN/FFmpeg-Builds, not a build this repository configures. What makes it LGPL is that neither --enable-gpl nor --enable-nonfree is present in its ffmpeg -buildconf. See THIRD-PARTY-NOTICES.md for the full reasoning, the pinned build identity, and where to obtain its corresponding source. It ships inside every package.

Two packages pack FFmpeg's binaries — FrameFlow.Native and FrameFlow.Native.Runtime — and both ship the operative licence texts alongside: LGPL-3.0, the GPL-3.0 it incorporates by reference, Apache-2.0 for the OpenCORE codecs inside avcodec, and LGPL-2.1 as the record of the upstream grant. FrameFlow.MotionClip packs no natives but receives them at publish time, so it ships the same texts. FrameFlow.Audio.OpenAL receives OpenAL Soft the same way and ships the LGPL-2.1 text that governs it.