Krackend.Sagas.Orchestrations.Runtime.Storage.EntityFramework.Mongo 6.4.1

Krackend

Modular backend building blocks for .NET services and orchestration runtimes.

Krackend runtime libraries, hosts, and samples target .NET 10. Applications consuming these packages require .NET 10 or later. The Roslyn analyzer retains its compatible netstandard2.0 target.

Version 6.4 highlights

v6.4.0 refreshes the orchestration Runtime dashboard, Statistics, Instances, History, and instance detail views with AG Grid, Apache ECharts, row inspectors, formatted payloads, and trace information. Control Plane and Runtime catalogs share configurable theming and improved navigation. Roadmap creation and paste actions use Zoorce speed dials, and a new draft opens directly.

Failure causes are stored as a Runtime projection. New failure events are projected as they occur, and historical events and their orchestration scopes are reconstructed gradually by the statistics job. Hosts must migrate the new failure-event and backfill-checkpoint storage and indexes; the SQL Server sample includes migrations. Statistics can be incomplete until backfill finishes. The release also repairs recovery of unfinished parallel tasks and caches artifacts during timeout repair passes.

Hosts continue to select and configure their own EF providers and migrations. The Control Plane SQL Server sample now includes the previously missing migration for designer concurrency tokens. See the v6.4.0 upgrade notes and Sagas configuration reference.

Version 6.3 highlights

v6.3.0 stores the effective lifetime and Idle durations and the UTC expiration deadline on each Runtime instance. The supervisor queries Running instances by the indexed deadline without loading artifacts, and recovery recalculates the deadline from the projected policy. Recoverable legacy instances are projected in lease-protected batches at startup.

Runtime hosts must apply the three instance fields and (Status, ExpiresOnUtc, Id) index before upgrading. The SQL Server sample includes a migration; hosts remain responsible for their EF provider and migrations. Stop older Runtime writers during the upgrade so they do not create unprojected instances after backfill. Custom instance repositories must implement IOrchestrationExpirationRepository. See the Sagas configuration reference and upgrade notes.

Version 6.2 highlights

v6.2.0 adds per-version orchestration lifetime and Idle policies, background supervision of existing instances, recovery for archived versions, and staged rollout and rollback controls. Versions without explicit settings inherit a 24-hour maximum lifetime and a 20-minute Idle window. Lifetime scans run at startup and daily by default, so expiration is detected on the next scan rather than exactly at the deadline.

Control Plane hosts upgrading from before v6.2 must add the nullable version fields MaxInstanceDurationSeconds and IdleDurationSeconds; the SQL Server sample includes a migration. Pigeon 4.2.0 cannot confirm broker consumer readiness by itself: staged rollout and rollback proceed after registration unless the host provides a readiness implementation. See the v6.2.0 upgrade notes.

Version 6.1.1 highlights

v6.1.1 sends a successful orchestration callback when Pigeon skips an already processed command, preserving its orchestration metadata without invoking the handler again. This callback does not restore the original business response payload; response validation still applies. The roadmap editor also groups paste actions in a floating menu and confirms copies with a non-blocking toast.

Version 6.1 highlights

v6.1.0 improves orchestration recovery, runtime monitoring, and the roadmap editor. Instance lists read projected version and progress data; the designer supports cloning and clipboard transfer of roadmap elements; and stalled instances can be reconciled through the runtime. The KnOwl adapter uses bounded server-side contract searches, while ButterMorph 2.1.5 provides the current designer session integration.

Hosts must apply their own EF migrations for the new runtime instance projection and Control Plane concurrency tokens. See the v6.1.0 upgrade notes and the storage guidance below. KnOwl contract autocomplete requires Control Plane 2.4.3 or later.

Version 6.0 highlights

v6.0.0 is a major orchestration release. It replaces the former direct version lifecycle with reviewed, applied deployment artifacts; adds portable orchestration-version packages; and hardens the runtime around durable dispatch cancellation, replay, response-timeout ownership, and recovery across replicas.

• Governed delivery. Drafts move through a review and explicit apply step before producing a deploy, deprecate, or archive artifact. Releases now target those immutable deployed artifacts, including direct and rollout delivery strategies.
• Portable design. Export an eligible orchestration version and import it into a selected orchestration as a new draft; its exported lifecycle state is informational and is never copied into the destination lifecycle.
• Durable operations. Mule-backed dispatches can be fenced, replayed, and cancelled durably. A response timeout is a single owned durable action, and instances that need reconciliation enter the Idle window before dead-lettering.
• Observable runtime. The Runtime UI exposes separate dashboard, instances, statistics, history, and artifact areas with scoped live updates. SignalR carries compact deltas so active views do not need to reload their data for each state change.
• Safer Pigeon outcomes. Terminal Pigeon consumer rejections become non-retryable orchestration failures, while client diagnostics can preserve full exception details for the matching task attempt.

Review the current Sagas API and configuration reference and the end-to-end walkthrough before upgrading a host. The release requires host-owned storage migrations for the new Control Plane lifecycle and Runtime durability fields.

Packages

Event sourcing packages:

dotnet add package Krackend.EventSourcing.Abstractions
dotnet add package Krackend.EventSourcing
dotnet add package Krackend.EventSourcing.EntityFrameworkCore
dotnet add package Krackend.EventSourcing.Projections
dotnet add package Krackend.EventSourcing.Analyzers
dotnet add package Krackend.EventSourcing.Testing

Saga orchestration core packages:

dotnet add package Krackend.Sagas.Orchestrations.Abstractions
dotnet add package Krackend.Sagas.Orchestrations.Contracts

Saga orchestration control-plane packages:

dotnet add package Krackend.Sagas.Orchestrations.ControlPlane
dotnet add package Krackend.Sagas.Orchestrations.ControlPlane.Application
dotnet add package Krackend.Sagas.Orchestrations.ControlPlane.Storage.EntityFramework
dotnet add package Krackend.Sagas.Orchestrations.ControlPlane.WebUI
dotnet add package Krackend.Sagas.Orchestrations.ControlPlane.Api

Saga orchestration runtime packages:

dotnet add package Krackend.Sagas.Orchestrations.Runtime
dotnet add package Krackend.Sagas.Orchestrations.Runtime.Storage.EntityFramework
dotnet add package Krackend.Sagas.Orchestrations.Runtime.Buffering.Mule
dotnet add package Krackend.Sagas.Orchestrations.Runtime.Messaging.Pigeon
dotnet add package Krackend.Sagas.Orchestrations.Runtime.Gossip.Redis
dotnet add package Krackend.Sagas.Orchestrations.Runtime.ButterMorph
dotnet add package Krackend.Sagas.Orchestrations.Runtime.WebUI
dotnet add package Krackend.Sagas.Orchestrations.Runtime.Api

Saga orchestration client and schema packages:

dotnet add package Krackend.Sagas.Orchestrations.Client
dotnet add package Krackend.Sagas.Orchestrations.Client.Messaging.Pigeon
dotnet add package Krackend.Sagas.Orchestrations.Client.Spider
dotnet add package Krackend.Sagas.Orchestrations.SchemaRegistry.Abstractions
dotnet add package Krackend.Sagas.Orchestrations.SchemaRegistry.KnOwl
dotnet add package Krackend.Sagas.Orchestrations.WebUI.Shell

Authentication integration package:

dotnet add package Krackend.Sagas.Orchestrations.WolfAuth

Event Sourcing

Krackend.EventSourcing provides a modular write-model runtime for event sourcing:

• event and state schema versioning
• expected-version appends
• paged stream reads
• state rehydration with reducers
• snapshots of state
• EF Core event store adapter
• configurable event envelope ids through IEventIdFactory, with ULID ids by default
• raw JSON event appends for centralized event stores
• runtime diagnostics
• Roslyn analyzers
• testing helpers

Minimal setup:

services.AddKrackendEventSourcing(options =>
{
    options.ScanAssemblyContaining<Program>();
    options.Stores.Add("customers", store => store.TableName = "CustomerEvents");
});

services.AddEventSourcedInitialStateFactory<CustomerState, CustomerInitialStateFactory>();
services.AddKrackendEntityFrameworkEventStore<AppDbContext>();

Storage ids and EF mapping:

Krackend uses string ULIDs for the EventId, SnapshotId, and CandidateId primary keys. The EF Core adapter stores them as portable 26-character text by default. The host can choose a compact binary representation and configure every persisted field without modifying its DbContext.

services.AddScoped<IEventIdFactory, HostEventIdFactory>();
services.AddKrackendEventSourcing();

services.AddKrackendEntityFrameworkEventStore<AppDbContext>(ef =>
{
    ef.Keys(keys => keys.UseUlidBytes("binary(16)"));
    ef.Events(events =>
    {
        events.ForAll((entity, store) =>
            entity.Property(x => x.Payload).HasColumnType("nvarchar(max)"));

        events.For("customers", (entity, store) =>
            entity.Property(x => x.StreamId).HasMaxLength(128));
    });
    ef.Snapshots(entity =>
        entity.Property(x => x.Payload).HasColumnType("nvarchar(max)"));
    ef.SnapshotCandidates(entity =>
        entity.Property(x => x.StreamId).HasMaxLength(128));
});

The provider package, column types, conversions, connection, and migrations remain owned by the host. IRawEventStore uses the same configured event entities and mappings as IEventStore. Binary key storage requires IEventIdFactory to keep returning valid ULID strings; custom non-ULID event ids should configure EventId separately through Events.

Usage:

[EventStream("customers")]
public sealed record RenameCustomer(string CustomerId, string Name)
    : IEventStreamCommand
{
    public string StreamId => CustomerId;
}

await customerService.ExecuteAsync(new RenameCustomer("customer-001", "New Name"));

Centralized raw event store usage:

var rawEventStore = provider.GetRequiredService<IRawEventStore>();

await rawEventStore.AppendRawAsync(
    streamName: "integration-events",
    streamId: "customers:customer-001",
    expectedVersion: ExpectedVersion.Any,
    events:
    [
        new RawEventData(
            "Customers.CustomerRenamed",
            "1.0.0",
            "{\"customerId\":\"customer-001\",\"name\":\"New Name\"}")
    ]);

See docs/event-sourcing.md for the full guide.

Samples:

• samples/Krackend.EventSourcing.Sqlite.Sample: typed event-sourced write model with SQLite.
• samples/Krackend.EventSourcing.Centralized.Sample: centralized raw JSON event store with SQLite.
• samples/Krackend.EventSourcing.Pelican.Sample: exploratory Pelican/template integration.
• samples/Krackend.Sagas.Orchestrations.RuntimeHost.Sample: host that mounts runtime storage, Mule buffering, Pigeon messaging, Redis gossip, ButterMorph, and Runtime WebUI.
• samples/Krackend.Sagas.Orchestrations.ControlPlaneHost.Sample: control-plane host that mounts design, distribution, team management, WebUI, artifact delivery endpoints, and EF migrations.
• samples/Krackend.Sagas.Orchestrations.RuntimeHost.Mongo.Sample: Mongo-backed runtime host that uses the MongoDB EF Core provider while keeping Mule buffering, Pigeon messaging, Redis gossip, ButterMorph, Runtime WebUI, and runtime APIs wired through the same orchestration packages.
• samples/Krackend.Sagas.Orchestrations.ControlPlaneHost.Mongo.Sample: Mongo-backed control-plane host that uses the MongoDB EF Core provider with design, distribution, team management, WebUI, artifact delivery endpoints, APIs, and seed data.

Runtime samples register the Azure Service Bus Pigeon adapter only when ConnectionStrings:AzureServiceBus or Pigeon:MessageBrokers:AzureServiceBus:ConnectionString is configured. With the default empty value, the hosts still start for local storage, distribution, and WebUI smoke testing without requiring a broker.

Sagas Orchestrations

Krackend Sagas Orchestrations is split into composable libraries so the runtime, control plane, transport adapters, client integrations, and UI modules can evolve independently:

• Krackend.Sagas.Orchestrations.ControlPlane* captures orchestration definitions, versions, releases, runtime nodes, credentials, and artifact delivery.
• Krackend.Sagas.Orchestrations.Runtime* consumes immutable artifacts, projects ingress configuration, runs durable Mule-backed work, dispatches transport-agnostic commands, tracks instances, and exposes runtime diagnostics.
• Krackend.Sagas.Orchestrations.Client* lets services start or answer orchestration work without changing business payloads. Client publishers pass the original business payload object to the selected transport adapter so the adapter owns serialization. Pigeon is one messaging adapter and Spider is a pipeline extension over the client core.
• The Spider client extension supports inline and deferred consumer execution. EmitEvent publishes new orchestration trigger events, while UseOrchestration keeps the service connected to an existing saga backchannel. Both can be composed in the same pipeline, allowing a consumer to emit one or more events and still report success or failure to the orchestration that invoked it. For deferred SquirrelBox consumers, orchestration metadata is restored from the inbox entry so success and failure callbacks still reach the runtime backchannel without wrapping or mutating the business payload.
• Failed client operations publish an OrchestrationExecutionResultMetadata record. By default it includes the normalized error code, message, CLR exception type, and complete Exception.ToString() diagnostic (including inner exceptions and stack trace). The runtime persists that diagnostic only on the matching task attempt and exposes it in the saga attempt detail with a copy action. Set IncludeExceptionDetails = false to redact all exception information, or set IncludeExceptionStackTrace = false to retain the message and type without the complete diagnostic. Treat complete diagnostics as sensitive operational data: secure the transport and runtime diagnostics access accordingly.
• Krackend.Sagas.Orchestrations.SchemaRegistry* keeps schema resolution provider-neutral, with KnOwl Control Plane available as the plug-in adapter for deployed ButterMorph contracts.
• Krackend.Sagas.Orchestrations.WebUI.Shell, ControlPlane.WebUI, and Runtime.WebUI provide Razor UI modules for host applications.
• Krackend.Sagas.Orchestrations.ControlPlane.Api and Runtime.Api expose optional REST endpoints over the same application/runtime services used by the WebUI modules.
• Krackend.Sagas.Orchestrations.WolfAuth adds optional WebUI authentication gates over host-owned WolfAuth/ASP.NET Core authentication. Krackend no longer ships its own subject, role, permission, or security storage packages.
• Control Plane and runtime ButterMorph integration is aligned with ButterMorph 2.1.5 for schema design, mapping, execution-condition authoring, runtime transforms, and runtime validations.

Web UI and themes

The Control Plane and Runtime UIs use the Zoorce application shell and controls. Runtime Dashboard and Statistics charts use Apache ECharts; the Runtime Instances and History tables use AG Grid Community. These libraries are loaded from their configured CDN URLs, so the browser must be able to reach them. Instance search, status filtering, sorting, and paging run in the configured storage provider; the grid does not load the complete instance history.

The Runtime dashboard also displays recorded failure events by category. New events are projected as they are recorded; existing transition history is reconstructed incrementally by the Mule statistics job, and the dashboard indicates while that reconstruction is incomplete. Counts represent failure events, including repeated failures, rather than unique failed instances. Apply the host's runtime EF migration before enabling the projection. The runtime model remains provider-independent; each host configures its own provider and schema lifecycle.

Dashboard cards count the current states of instances started in the selected period. Their small charts show recorded state changes for that same cohort, not historical snapshots of the card counts; a single instance can change state more than once. The activity chart uses the same cohort and reports transitions recorded during the selected period. Failure causes are a separate event view covering all failure events in the period, including those belonging to older instances. Statistics recorded before the projection was enabled may not contain transition history. Live dashboard totals refresh from committed statistics projections with coalesced notifications.

Configure the light and dark palettes when registering the UI in each host. The same configuration supplies the Zoorce shell and, in the Control Plane, the embedded ButterMorph designers. Existing top-level color properties are aliases for the light palette.

using Krackend.Sagas.Orchestrations.WebUI.Shell;

builder.Services.AddOrchestratorControlPlaneWebUI(options =>
{
    options.Theme.Title = "Krackend";
    options.Theme.Mode = OrchestratorWebUIThemeMode.Light;
    options.Theme.Light.PrimaryColor = "#0b3d91";
    options.Theme.Light.SurfaceColor = "#ffffff";
    options.Theme.Light.ElevatedColor = "#f7faff";
    options.Theme.Light.PrimarySoftColor = "#e4edff";
    options.Theme.Light.SuccessColor = "#087a57";
    options.Theme.Light.SuccessSoftColor = "#ddf4e9";
    options.Theme.Dark.PrimaryColor = "#73a8ff";
    options.Theme.Dark.SurfaceColor = "#0d1b2f";
    options.Theme.Dark.ElevatedColor = "#172a42";
    options.Theme.Dark.PrimarySoftColor = "#253d62";
    options.Theme.Dark.SuccessColor = "#55d99c";
    options.Theme.Dark.SuccessSoftColor = "#173d30";
});

// In the separate runtime host:
builder.Services.AddOrchestratorRuntimeWebUI(options =>
{
    options.Theme.Mode = OrchestratorWebUIThemeMode.Light;
    options.Theme.Light.PrimaryColor = "#0b3d91";
    options.Theme.Dark.PrimaryColor = "#73a8ff";
});

When using AddOrchestratorControlPlane instead of the standalone UI registration, set options.Theme on that composition. Configure both palettes with sufficient contrast; the host-provided initial mode is independent of the user's light/dark selection.

Each palette can also configure PrimaryForegroundColor, GlowColor, warning, danger and information colors and their soft variants, five data visualization colors, code control colors, shadow, and overlay. Optional elevated, soft-accent, glow and primary-foreground colors are derived from the configured surface and primary color when omitted. Legacy top-level properties remain aliases for Light. ButterMorph receives the supported subset of both palettes; its UI is not restyled by Krackend.

Spider event publishing:

Use EmitEvent when a service should publish an orchestration trigger event without reporting that publication as a saga reply. Keep UseOrchestration on the same pipeline when the service is also handling a command from an existing saga and must answer the runtime backchannel.

app.MapPost("/sales", async (CreateSaleRequest request, ISaleService sales) =>
{
    await sales.CreateAsync(request);
})
.EmitEvent(request => new SaleCreated(request.SaleId), "events.sales.sale.created")
.UseOrchestration();

EmitEvent uses the same routing pattern as UseOrchestration, including conditional routes and optional payload transforms. Multiple EmitEvent registrations can run before the final UseOrchestration hook when a service needs to raise follow-up business events and still complete the command that the saga sent.

Design validation:

Orchestrator component keys are internal identifiers and use alphanumeric segments separated by dots or underscores, starting with a letter. They preserve the casing captured by the host/UI and are not normalized to lowercase by Krackend.

Messaging topics are external broker addresses, not orchestrator keys. Event trigger topics, task command topics, and compensation topics are stored as captured after trimming and can follow the naming rules of the selected transport or provider, including dashes, underscores, dots, and uppercase characters.

Enable flags are part of the deployed orchestration shape. Disabled triggers do not start the orchestration, disabled stages are skipped by the runtime, and disabled tasks are not dispatched. This is useful for staged rollouts or temporarily removing a branch from execution without deleting the design history.

Orchestration metadata:

Krackend propagates transversal metadata through transport metadata so business payloads stay focused on business data. Runtime command dispatches include Krackend.Sagas.Orchestrations.Message.Metadata for the current runtime backchannel and flat propagated entries such as Krackend.Sagas.Orchestrations.Trigger.Metadata, audit.context, and security.context. The runtime no longer duplicates propagated metadata inside the reserved Krackend.Sagas.Orchestrations.Propagation.Metadata envelope when publishing commands; that envelope is only understood as a legacy inbound shape for compatibility.

Krackend.Sagas.Orchestrations.Trigger.Metadata is the canonical trigger metadata entry. It carries values such as correlation id, trace id, event id, event type, idempotency key, aggregate type, and causation id. The runtime uses the trigger correlation id as the saga correlation id when it is available, and still accepts the legacy trigger_metadata key when reading older messages.

Client consumers forward incoming propagation metadata when publishing follow-up messages. Non-reserved metadata entries such as audit.context and security.context are attached back to the outgoing transport metadata, and Krackend.Sagas.Orchestrations.Trigger.Metadata is forwarded as the canonical trigger context. Reserved Krackend metadata keys are not blindly forwarded.

When a client publishes a new trigger through EmitEvent, the outgoing trigger context is produced by IOrchestrationTriggerMetadataAccessor. That event receives a fresh Krackend.Sagas.Orchestrations.Trigger.Metadata entry for the new trigger publication. If the service was already executing inside a saga, the previous orchestration trace context is preserved separately under Krackend.Sagas.Orchestrations.Origin.Metadata so the new event can be traced back to the saga that emitted it without making the old saga correlation id the principal trigger correlation id of the new event.

Runtime trigger promotion is idempotent when a stable start key is available. The Mule adapter maps the durable action deduplication key, or the durable action id when no deduplication key exists, into the runtime start idempotency key. The core runtime also falls back to the trigger metadata idempotency key. Retried trigger actions therefore continue the already-promoted OrchestrationInstance instead of creating a second saga instance after a partial start, timeout, or action re-execution. Hosts using Entity Framework runtime storage should apply a migration for the nullable StartIdempotencyKey column and its unique non-null index.

With Mule 2.2, the runtime records cancellation intent before requesting cancellation of a pending durable dispatch. Reconciliation retries deferred cancellation after restarts or provider failures; it does not interrupt an action already claimed by a worker. Late callbacks and configured task retries retain their existing behavior. Custom runtime storage implementations should implement IOrchestrationDispatchCancellationIntentRepository and ITaskDispatchCancellationRepository to support recovery of cancellation intents. Hosts own their database migrations and must update the runtime model for OrchestrationInstance.DispatchCancellationPending and the TaskDispatch.PhysicalCancellation* fields, and the Mule model for DurableAction.CancelledOnUtc and MuleReplay.CancelledOnUtc. The SQL Server runtime sample includes migrations for these changes; other providers must generate their own.

Awaited task responses now own a single durable timeout action. Dispatch records the response deadline and its Mule reference after committing the WaitingResponse state; a callback clears that reference and cancels the timer. If a callback and timer race, both run under the orchestration lease and the stale side becomes a no-op. The runtime still accepts late callbacks according to its existing reconciliation policy. Entity Framework hosts must migrate TaskExecution.ResponseDeadlineUtc, TimeoutActionId, and TimeoutActionLane and add the corresponding waiting-response indexes; the SQL Server runtime sample includes that migration.

Runtime task dispatch is adapter-based. Initial dispatch, retry dispatch, compensation dispatch, and runtime artifact compatibility validation resolve an installed task adapter by TaskKind instead of hardcoding the messaging transport in the engine. The built-in runtime registers the Messaging adapter by default; other task kinds must provide and register an ITaskRuntimeAdapter before artifacts using that kind can be deployed or dispatched.

Retry policies are explicit error-code allowlists. When MaxRetries is greater than zero, RetryableErrorCodes must contain at least one non-empty code; an empty list means the runtime will not retry any failure. Runtime callbacks can also set IsRetryableCandidate = false to suppress retry even when the reported error code appears in the allowlist.

Execution policies and extension bundles:

Orchestration artifacts use an extension-ready schema. Existing linear artifacts are migrated at runtime to schema v2, which adds built-in capability metadata, required external capabilities, required bundles, and execution policy slots without changing the behavior of existing messaging orchestrations.

The runtime resolves execution policy hierarchically: environment defaults, runtime-node overrides, orchestration overrides, stage overrides, and finally task overrides. The selected policy is validated against runtime node capabilities and stored on task attempt/dispatch metadata as ResolvedExecutionPolicy. The default provider is built-in-local, so existing tasks continue to execute through the installed in-process task adapters.

External bundles are explicit runtime state. Artifacts that require an external bundle or capability are accepted only when the runtime node has an activated RuntimeExtensionPackage with the matching bundle id, extension key, semantic version, checksum when provided, and manifest capability. The in-memory repository is suitable for tests or simple hosts; Krackend.Sagas.Orchestrations.Runtime.Storage.EntityFramework persists activation state in Runtime.RuntimeExtensionPackages for durable multi-replica hosts.

That activation state is part of the shared Entity Framework runtime model, so relational hosts and the MongoDB EF provider used by the Mongo runtime sample validate bundles through the same repository contract.

Hosts can configure runtime execution defaults through Runtime:Execution or RuntimeExecutionOptions:

services.Configure<RuntimeExecutionOptions>(options =>
{
    options.EnvironmentPolicy = new ExecutionPolicyArtifact
    {
        DefaultProviderKey = ExecutionConstants.BuiltInLocalProvider,
        AllowedProviderKeys = [ExecutionConstants.BuiltInLocalProvider],
        RequireSandboxForExternalExtensions = true
    };
    options.RuntimeNodeCapabilities = RuntimeNodeCapabilitiesArtifact.LocalDefaults;
});

Custom execution providers can be registered by hosts through the runtime builder:

services
    .AddKrackendOrchestrationsRuntime()
    .AddExecutionSandboxProvider<KubernetesExecutionSandboxProvider>();

Recoverable runtime failures:

When a task exhausts its configured retries, times out, or leaves the orchestration unable to advance, the runtime moves the instance into DeadLettered when the failure can still be reviewed by an operator. DeadLettered is not a broker queue; it is an explicit durable state that says the saga stopped, preserved its context, and can be resumed after the underlying issue is corrected. Failed and Aborted are reserved for outcomes that should not continue automatically.

The Runtime API exposes recovery operations under the configured runtime API prefix, /api/v1/runtime by default:

POST /api/v1/runtime/instances/{instanceId}/replay
Content-Type: application/json

{ "payload": "{...optional replacement payload...}" }

POST /api/v1/runtime/instances/{instanceId}/abort
Content-Type: application/json

{ "reason": "External order was cancelled by support." }

Replay finds the latest failed or timed-out task first and dispatches it again through the configured task adapter. If no failed task exists but a stage failed, replay restarts that stage. If neither exists, the runtime re-enters the forward engine with the preserved instance snapshot or the optional replacement payload. Abort marks the instance as Aborted and records the operator reason in metadata and transitions.

Late callbacks are accepted conservatively. If an earlier timeout attempt eventually reports success after the instance already moved to DeadLettered, the task execution can be completed so idempotent downstream systems can reconcile correctly, while the instance remains in the operator-visible stopped state until replay or abort.

Runtime operations and disaster recovery:

The runtime evaluates registered dependency probes and exposes an admission controller before accepting ingress, callbacks, dispatch, timeout processing, startup projection, recovery, and reconciliation work. Primary persistence is a critical dependency: if the main database is unavailable, the runtime moves to Closed and stops accepting new work so brokers can retain messages or upstream systems can retry through their own outbox/retry strategy. Optional dependencies such as Redis gossip degrade the runtime instead of closing it; the durable database remains the source of truth.

Transport and infrastructure adapters receive operational state changes through IRuntimeDegradationHandler. The engine stays transport-agnostic and the adapter decides how to pause, resume, nack, defer, or log according to the selected protocol. The built-in Pigeon adapter listens for closed/reopened states without hardcoding broker behavior into the runtime engine.

Runtime reconciliation runs in the background and can also be invoked through IOrchestrationRuntimeReconciler. It repairs only durable inconsistencies: unresolved physical dispatch cancellations, persisted-but-unconfirmed dispatches, compensation delivery intents, and awaited tasks missing their durable timeout reference. It does not enumerate every healthy Running or Waiting saga or re-enter the saga engine globally. This preserves restart and failover recovery without turning normal runtime load into repeated work.

The lifetime supervisor separately checks running instances against their deployed version's maximum lifetime. Versions without a policy inherit the runtime default of 24 hours; the default idle window is 20 minutes. Both values can be set on a draft roadmap in the Control Plane; a maximum of zero disables expiration for that version. The supervisor scans on startup and once per day by default, so an instance started before an upgrade can enter Idle on the first scan. Hosts can change the defaults and scan interval through Runtime:Timeouts. Saga and task recovery start a new lifetime window; valid late callbacks do not extend the existing idle deadline.

Expiration is detected by the next scan, so it is not an exact deadline: with the default daily interval, detection can take up to another day. The Entity Framework and in-memory repositories query Running instances directly by projected ExpiresOnUtc in pages; the supervisor does not load artifact policies. Custom instance repositories implement IOrchestrationExpirationRepository to support expiration scans and legacy policy projection.

Hosts own their storage migrations. Add the two nullable Control Plane version fields and the three Runtime instance fields plus the (Status, ExpiresOnUtc, Id) index before deploying updated hosts; SQL Server sample migrations are included. At startup, a separate paged, lease-protected backfill projects policies for recoverable legacy instances and caches each artifact within that run. Unavailable or leased instances are retried on the next lifetime scan. A null maximum indicates an unprojected legacy instance; zero means expiration is disabled. Stop older writers before completing the upgrade, because they do not populate these fields. Runtime defaults are captured during projection and do not retroactively alter already-projected instances.

The Control Plane SQL Server sample includes the AddDesignConcurrencyTokens migration for StageDefinitions, TaskDefinitions, and TriggerBindings. The lifetime migration does not add those columns. Hosts with existing databases must apply their own migration and follow the concurrency-token upgrade guidance below.

For a staged Pigeon rollout or archived recovery, the consumer acknowledgement mode must be OnHandlerSuccess. A delivery with no active matching ingress fails through the execution pipeline before Pigeon requests retry, so an idempotency inbox does not mark it complete or race with redelivery. The transport must support retry/redelivery. Manual and OnReceive are rejected for staged connector preparation.

Pigeon 4.2.0 does not expose a transport-independent confirmation that an asynchronously started consumer is accepting deliveries. Staged rollouts and archived recoveries therefore proceed after consumer registration by default; Ready does not guarantee the broker accepted a newly registered consumer, and a later startup failure cannot automatically roll back the version. Use durable broker queues and monitor consumer health. Hosts that can independently verify their transport may register an IPigeonConsumerReadiness implementation to require confirmation; the optional wait defaults to 60 seconds and can be changed with Runtime:Pigeon:ConsumerReadiness:TimeoutSeconds. Registration alone does not satisfy that confirmation contract. The Pigeon readiness requirement describes the capability needed for a general solution.

Recovering an instance of an archived version prepares its backchannel before dispatching work and keeps the version's triggers disabled. The version remains archived. After its last active instance and mutation lease finish, the lifetime scan retires the restored backchannel; each replica closes its stale local connector on its next scan. The scan runs at startup and daily by default, so connector cleanup is not immediate. Entity Framework storage uses the existing generation and instance lease to coordinate archive, recovery and retirement. Automatic retirement requires Runtime:Timeouts:Enabled and Runtime:Coordination:Enabled, transactional storage, and an instance repository that reports active leases. Custom storage can provide IArchivedRecoveryRetirementRepository, IArchivedRecoveryBatchRepository and IOrchestrationArtifactActivityBatchReader; otherwise the cleanup is skipped. Artifact activity is checked in batches rather than scanning the instances table once per archived version.

The Entity Framework runtime storage requires a provider that supports server-side ExecuteUpdate for atomic instance leases. The EF Core InMemory provider does not support that operation; use Krackend's in-memory runtime storage for non-durable tests, or an EF provider that supports atomic updates. The host remains responsible for choosing and configuring its provider.

Staged rollout also requires a durable projection scheduler implementing IRuntimeDurableRolloutProjectionScheduler; the Entity Framework scheduler implements it. A conflicting rollout is rejected before changing trigger ownership when this capability is unavailable.

The Runtime UI's rollback action prepares an earlier active version and switches its triggers back after consumer registration. The active versions being replaced also have their exclusive triggers disabled; their backchannels remain available for in-flight instances. Its history records whether the requested switch is pending, ready or failed.

{
  "Runtime": {
    "Operations": {
      "Enabled": true,
      "ScanIntervalSeconds": 5
    },
    "Reconciliation": {
      "Enabled": true,
      "ScanIntervalSeconds": 15,
      "BatchSize": 100
    },
    "Timeouts": {
      "DefaultInstanceLifetimeSeconds": 86400,
      "DefaultIdleDurationSeconds": 1200,
      "InstanceLifetimeScanIntervalSeconds": 86400,
      "InstanceLifetimeBatchSize": 100
    }
  }
}

Compensation:

Tasks and event triggers can define compensating tasks. The runtime evaluates the compensation execution condition, applies the compensation transformation, and dispatches the transformed compensation request through the task adapter abstraction. This keeps compensation transport-agnostic and lets messaging, or another installed task kind, own its own dispatch behavior.

ButterMorph compensation contexts include the data that is technically available at that point in the saga. Task compensation can use the trigger payload and metadata, forward task requests and replies up to the task being compensated, the failed task request when available, and previous compensation replies. Trigger compensation intentionally exposes only the selected event trigger payload, trigger metadata, and transversal metadata, because forward tasks may never have executed when trigger compensation runs. This lets a compensation step reverse the paired forward task and still emit additional compensating side effects when the design requires them.

The Control Plane WebUI opens ButterMorph directly for compensation execution conditions and transformations. Trigger compensation is edited in its own tab with the same task configuration surface used by regular tasks, including timeout, retry, and on-error policy settings.

The ButterMorph package family is versioned centrally by Krackend so design-time snapshots and runtime execution stay on the same mapping engine.

The mapping and execution-condition designers use isolated in-memory ButterMorph sessions. A multi-replica Control Plane must route an editor's requests to the same replica. Sessions close when the embedded editor closes and abandoned sessions expire under ButterMorph's retention policy. The payload schema designer used for metadata has a separate lifecycle.

Designer saves compare the schema and design revisions last loaded by the editor with the current values and report conflicts without replacing the local draft. Changes to the same task, stage, or trigger are also protected by an EF optimistic concurrency token during the write. A concurrent schema change in another record is still subject to the interval between checking the schema and committing the design.

The Control Plane EF model requires a non-null ConcurrencyToken string of up to 32 characters on TaskDefinitions, StageDefinitions, and TriggerBindings. Before deploying to an existing relational database, the host must add these columns and initialize existing rows to the empty string through its EF migration; the SQL Server sample includes AddDesignConcurrencyTokens. MongoDB hosts must likewise initialize the field on existing documents before deploying. Drain older Control Plane instances during this upgrade: instances that do not update the tokens cannot participate in the concurrency check. EF manages the tokens independently of the database provider after the upgrade.

Custom Control Plane storage implementations must implement the conditional designer-save operations on the task, stage, and trigger repositories. Their default implementations reject the save rather than silently bypassing the concurrency check.

Authentication model:

Authentication belongs to the host. Krackend libraries do not configure Entra ID, JWT bearer, cookies, API keys, IdentityServer, Auth0, Keycloak, or any concrete provider. The host configures WolfAuth and ASP.NET Core authentication, then optionally enables Krackend's WolfAuth integration so the WebUI redirects anonymous browser users to a login shell and requires an authenticated ClaimsPrincipal.

using Krackend.Sagas.Orchestrations.WolfAuth;
using WolfAuth.AspNetCore;

builder.Services
    .AddAuthentication("HostScheme")
    .AddJwtBearer("HostScheme", options =>
    {
        // Host-owned authentication configuration.
    });

builder.Services.UseWolfAuth(builder.Configuration.GetSection("Krackend:WolfAuth"), options =>
{
    options.ApplicationName = "Krackend";
    options.ChallengeSchemes.Add("HostScheme");
});

app.UseAuthentication();
app.UseWolfAuth();
app.UseKrackendWolfAuthLoginGate();
app.UseAuthorization();
app.MapKrackendWolfAuthEndpoints();

UseWolfAuth is only Krackend's product integration point. It does not create users, roles, permissions, or databases. If the package is not registered, the WebUI shell renders without login/logout controls and Krackend leaves authentication completely to the host.

Entity Framework storage provider model:

The *.Storage.EntityFramework packages define portable EF Core storage mappings and repository implementations, but they do not choose SQL Server, PostgreSQL, SQLite, or any other database provider. The host owns the provider package, connection string, migrations assembly, and any provider-specific model details.

builder.Services.AddOrchestratorRuntimeStorageEntityFramework(
    db => db.UseSqlServer(
        builder.Configuration.GetConnectionString("Runtime"),
        sql => sql.MigrationsAssembly(typeof(Program).Assembly.GetName().Name)),
    storage => storage.ConfigureModel = modelBuilder =>
    {
        // Optional host-owned SQL Server details, such as filtered indexes
        // or provider-specific column types used by this host's migrations.
    });

The runtime also offers optional provider packages. The host may add Krackend.Sagas.Orchestrations.Runtime.Storage.EntityFramework.SqlServer and call modelBuilder.UseKrackendSqlServerOptimizations() inside ConfigureModel. The filtered statistics backfill index is enabled by default; pass options => options.FilterStatisticsBackfillIndex = false to disable it, or override the EF index configuration later in the same callback. Configure custom column names before applying the package's defaults. The host owns, generates, reviews, and applies the corresponding migrations.

The same host customization pattern is available for Control Plane storage.

Distribution credential protection:

Control Plane runtime-node credentials and Runtime design-node credentials are durable records. The Entity Framework storage adapters therefore persist the ASP.NET Core Data Protection key ring in the same storage model by default, under Distribution.DataProtectionKeys for Control Plane and Runtime.DataProtectionKeys for Runtime. This keeps protected distribution credentials decryptable across restarts, rollouts, and multiple replicas.

Hosts can keep the default database-backed key ring, set a stable Data Protection application name, or replace/harden key storage through the storage options:

builder.Services.AddOrchestratorRuntimeStorageEntityFramework(
    db => db.UseSqlServer(builder.Configuration.GetConnectionString("Runtime")),
    storage =>
    {
        storage.DataProtectionApplicationName = "krackend-runtime-prod";
        storage.ConfigureDataProtection = dataProtection =>
        {
            // Optional host-owned Data Protection hardening such as a certificate
            // or another persistent key repository.
        };
    });

The same options are available on AddOrchestratorControlPlaneStorageEntityFramework. If a host disables database key persistence or replaces it with another repository, that repository must be durable and shared by every replica that needs to decrypt existing distribution credentials. Credentials protected with a key that has already been lost cannot be recovered and must be reimported or regenerated.

MongoDB sample hosts are included to validate that provider choice belongs to the host:

builder.Services.AddOrchestratorRuntimeStorageEntityFramework(
    db => db.UseMongoDB(
        builder.Configuration.GetConnectionString("Runtime")!,
        builder.Configuration.GetValue("Mongo:DatabaseName", "KrackendRuntime")),
    storage => storage.ConfigureModel = model => model.UseKrackendMongoRuntimeModel());

The runtime host must reference Krackend.Sagas.Orchestrations.Runtime.Storage.EntityFramework.Mongo and import its namespace for UseKrackendMongoRuntimeModel. Existing Mongo runtime hosts must add that package and configure it when upgrading: Mongo-specific key mappings no longer live in the common EF adapter. Host model configuration may continue after the call to customize other mappings. The Mongo sample explicitly calls EnsureCreatedAsync, which creates model indexes and can add new indexes to an existing database. Existing indexes are not modified or deleted automatically; changing or removing them requires an explicit host-controlled database operation. Use the Mongo provider's named HasIndex overload for custom index names.

The Mongo samples expect MongoDB to run as a replica set because the orchestration runtime uses transactions for durable state changes.

REST APIs can keep an optional global policy and also opt into granular product policies:

app.MapKrackendOrchestrationsControlPlaneApi(options =>
{
    options.Authorization.ReadPolicy = "ControlPlaneReaders";
    options.Authorization.DesignWritePolicy = "DesignWriters";
    options.Authorization.ReleaseExecutePolicy = "ReleaseOperators";
    options.Authorization.SecurityManagePolicy = "TeamManagers";
});

app.MapKrackendOrchestrationsRuntimeApi(options =>
{
    options.Authorization.ReadPolicy = "RuntimeReaders";
    options.Authorization.InstancesReadPolicy = "RuntimeInstanceReaders";
    options.Authorization.ManagePolicy = "RuntimeManagers";
    options.Authorization.ArtifactApplyPolicy = "RuntimeArtifactOperators";
});

The WebUI and REST API are sibling entry points over the same application/runtime services:

WebUI -> Application/Runtime services
API   -> Application/Runtime services

KnOwl schema registry setup for a design/control-plane host:

builder.Services.AddKrackendKnOwlSchemaRegistry(options =>
{
    options.BaseUri = new Uri("https://knowl-control-plane.local");
    options.ProviderKey = "knowl";
});

The KnOwl adapter reads deployed contracts from the KnOwl Control Plane. Event bindings resolve one event artifact, while command bindings can resolve the command request and reply artifacts together so orchestration artifacts keep immutable snapshots for validation, transformation, runtime dispatch, and response handling.

Hosts can override the KnOwl catalog route templates when their Control Plane exposes a custom API shape, and can register IKnOwlControlPlaneCatalogRequestConfigurator when catalog requests need host-owned authentication headers.

Contract autocomplete requires KnOwl Control Plane 2.4.3 or later and a storage provider that supports deployed-contract search. The adapter sends the search text, contract type, and result limit to contracts/artifacts/search (configurable with SearchPath); search results contain summaries rather than schema documents. Full schemas are retrieved through the existing exact or latest contract routes when a contract is resolved. KnOwl returns at most 100 results per search and matches an artifact ID as a complete GUID. A server without bounded search is reported as unavailable rather than falling back to downloading the entire catalog.

Minimal runtime host setup:

builder.Services.AddOrchestratorRuntimeWebUI(options =>
{
    options.RoutePrefix = "runtime";
});

builder.Services.AddOrchestratorRuntimeStorageEntityFramework(options =>
{
    options.UseSqlServer(builder.Configuration.GetConnectionString("Runtime"));
});

builder.Services
    .AddKrackendOrchestrationsRuntime()
    .AddPigeon(builder.Configuration)
    .AddMule(mule =>
    {
        mule.UseEntityFrameworkCore<RuntimeDbContext>();
    }, runtimeMule =>
    {
        runtimeMule.Profile = KrackendMuleRuntimeProfile.Balanced;
    });

builder.Services.AddKrackendOrchestrationsRuntimeButterMorph();

app.MapOrchestratorRuntimeDistributionEndpoints();
app.MapKrackendOrchestrationsRuntimeApi();
app.MapOrchestratorRuntimeReactiveHub();

Mule capacity profiles are runtime-owned, so hosts do not need to reproduce Krackend's internal lane topology. Balanced is the default; Development targets low-resource local work; HighThroughput raises the total reader and execution budget for provisioned production hosts; and Custom accepts explicit total budgets. Krackend partitions that budget across trigger intake, backchannels, dispatch, response timeouts, statistics, and artifact lifecycle lanes so a busy dispatch lane cannot starve callbacks or timeout processing.

{
  "Runtime": {
    "Mule": {
      "Profile": "HighThroughput"
    }
  }
}

For Custom, set TotalWorkerCount and TotalMaxDegreeOfParallelism; only use per-lane overrides when there is a measured reason. Runtime response timers and scheduled dispatches are persisted in Mule and have an exact wake-up notification. This avoids a periodic query over every waiting saga while preserving restart recovery from durable storage.

When multiple independent runtime stores share Redis, give each one a distinct Runtime:Mule:FastLane:KeyPrefix. Replicas that share the same runtime database must use the same prefix; separate environments, tenants, or isolated runtime databases must not. The sample defaults to krackend:runtime and hosts can override it without reproducing the runtime lane configuration.

Pigeon adapter configuration:

Pigeon acknowledgements are a host transport concern. A runtime or service host that does not explicitly complete deliveries must select OnHandlerSuccess; Pigeon's default is Manual and will otherwise leave successfully handled Azure Service Bus deliveries eligible for redelivery when their lock expires. The runtime samples use this safe default:

{
  "Pigeon": {
    "ConsumerExecution": {
      "AcknowledgementMode": "OnHandlerSuccess"
    }
  }
}

Use Manual only when the host's consumer deliberately calls ConsumeContext.CompleteAsync() or FailAsync(...) for every terminal path.

Krackend keeps the regular Pigeon global settings callback and also exposes Pigeon's full IPigeonServiceBuilder so hosts can configure serializer options, custom serializers, route interceptors, or other Pigeon features without bypassing the Krackend adapter:

using System.Text.Json;

builder.Services
    .AddKrackendOrchestrationsRuntime()
    .AddPigeon(
        builder.Configuration,
        settings =>
        {
            settings.SetDomain("orders-runtime");
        },
        pigeon =>
        {
            pigeon.ConfigureJsonOptions(options =>
            {
                options.PropertyNamingPolicy = JsonNamingPolicy.CamelCase;
            });
        });

The same full-builder overload is available for Krackend.Sagas.Orchestrations.Client.Messaging.Pigeon.

Minimal control-plane host setup:

builder.Services.AddOrchestratorControlPlane(options =>
{
    options.AdminRootPath = "admin";
    options.ConfigureStorage = db =>
        db.UseSqlServer(builder.Configuration.GetConnectionString("ControlPlane"));
});

app.MapOrchestratorArtifactDeliveryEndpoints();
app.MapKrackendOrchestrationsControlPlaneApi();

See docs/krackend-sagas-api-configuration.md for the current Sagas package, API, configuration, endpoint, Redis, storage, and host-integration reference. See docs/sagas-orchestrations-end-to-end.md for the Orchestrator walkthrough with models, lifecycle steps, distribution flow, runtime execution, playbooks, risks, and diagnostics. The older docs/sagas-orchestrations.md includes historical package names and examples.

Release

Packages are produced only by explicit dotnet pack or by the Build and Release workflow. The repository .release file contains the next release tag, for example v3.0.0; when that marker changes on main, the workflow validates the changelog section, creates the release branch/tag, packs all source libraries, and publishes the NuGet artifacts.

Event Sourcing Testing

Krackend.EventSourcing.Testing provides reducer/decider helpers and DI-friendly services for testing event sourcing flows without a real event store. It is meant for application tests, package adapters, and external test hosts that need to assert event sourcing behavior without booting EF Core, SQL Server, SQLite, or a production event store.

services.AddKrackendEventSourcingTesting();

var eventStore = provider.GetRequiredService<IEventSourcingTestEventStore>();
var assertions = provider.GetRequiredService<IEventSourcingTestAssertions>();
var stream = EventStreamReference.Create("customers", customerId);

await eventStore.AppendAsync(stream, [new CustomerCreated(customerId)]);

await assertions.ShouldHaveEventAsync<CustomerCreated>("customers", customerId);
await assertions.ShouldHaveVersionAsync("customers", customerId, 1);

Append with expected-version behavior and metadata:

await eventStore.AppendAsync(
    stream,
    ExpectedVersion.NoStream,
    [new CustomerCreated(customerId)],
    new Dictionary<string, object?>
    {
        ["correlation-id"] = correlationId
    });

Available assertions include:

• stream existence
• event type
• event order
• stream version
• metadata
• serialized payload

Failure simulation:

eventStore.FailNextAppendWithConcurrencyConflict(stream);

External test host integrations can use:

services.AddKrackendEventSourcingTestingAdapter();

That adapter-friendly registration allows wrappers such as:

testHost.UseKrackendTesting();