NetworkInspector.Core 0.20.0

NetworkInspector.Core

The parsing engine of NetworkInspector: turn raw frame bytes into a typed field tree, record what you need on the way, and query across packets. Pair it with NetworkInspector.Protocols for the built-in dissectors.

Highlights

• Allocation-aware parsing. Packets store a flat field tree in small chunks from a thread-local slab; TryParse can recycle a packet without allocating field storage again, and lazy protocol subtrees are only expanded when you look at them.
• Parse only what you need. FieldTreeMode.Skip decodes without building a tree, for value caches, indexes, and field watches that do not need the whole packet.
• Typed values. Every field carries a typed FieldValue (integers, addresses, timestamps, strings, bytes), a display text, and its FieldInfo metadata.
• Cross-packet presence index. PacketIndex records which protocols and fields occur in which packets in Roaring bitmaps, for fast filtering over millions of packets.
• Columnar value caches. ValueCache records selected field values into RAM columns during the first parse.
• Stateful protocols done right. TCP stream reassembly, the generic FragmentReassembler<TKey> (IPv4, IPv6, and every later fragmenting protocol, configured by FragmentReassemblyOptions with limits, alignment, overlap, eviction, restart, and gap policies), and per-packet effect stores make re-parsing any packet, on any thread, give the same result as the first pass. Every call of the reassembler returns a FragmentOutcome that names what happened, so a refused fragment is a reported reason and not a silent null. IngestGate is the one first-parse-versus-replay decision a stateful protocol needs: IsReplay(id) for the replay path, Enter(id) for the scope of the first parse, and the outermost scope raises the watermark that publishes the packet's recorded effects.
• Results instead of exceptions. Parsing returns bool plus a ParseFailure code; malformed input becomes packet.error fields, not crashes.
• Declarative dissectors. The bundled Roslyn source generator writes the registration code for your own protocols.

Install

dotnet add package NetworkInspector.Core
dotnet add package NetworkInspector.Protocols

NetworkInspector.Values and the source generator come with Core.

Quick Start

Parse one Ethernet / IPv4 / UDP frame and print its fields:

using NetworkInspector.Core;
using NetworkInspector.Core.Fields;
using NetworkInspector.Core.Ids;
using NetworkInspector.Core.Infos;
using NetworkInspector.Core.Settings;
using NetworkInspector.Protocols;
using NetworkInspector.Values;

// Ethernet II / IPv4 192.168.1.1 -> 192.168.1.2 / UDP 12345 -> 53, no payload (42 bytes)
byte[] rawBytes = Convert.FromHexString(
    "001122334455" + "66778899AABB" + "0800" +    // Ethernet: destination, source, EtherType
    "4500001C" + "12344000" + "4011A549" +        // IPv4: version/length, id/flags, TTL/protocol/checksum
    "C0A80101" + "C0A80102" +                     // IPv4: source, destination
    "30390035" + "00080000");                     // UDP: ports, length, checksum

// Build the stack once and reuse it; it owns and disposes the SettingsManager.
StackBuilder builder = new(new SettingsManager(), new FrameInterfaceRegistry());
builder.RegisterStandardProtocols();
using Stack stack = builder.Build();

Frame frame = Frame.Create(
    0,
    Timestamp.FromSecs(1_700_000_000),
    rawBytes,
    LinkType.Ethernet,
    FrameInterfaceId.Invalid,
    stack.FrameInterfaceRegistry).Value;

ParseOptions options = new(); // build a field tree
if (!Packet.TryParse(0, stack, frame, in options, out Packet? packet, out ParseFailure failure)
    || packet is null)
{
    throw new InvalidOperationException(failure.ToString());
}

Console.WriteLine(packet.Info);
foreach (Field field in packet.IterFieldsDfs(materialize: true))
{
    FieldInfo? info = field.FieldInfo;
    Console.WriteLine($"{info?.Name}: {field.Value}");
}

Output (abridged):

Src Port: 12345, Dst Port: 53
frame.len: 42
eth.dst: 00:11:22:33:44:55
eth.src: 66:77:88:99:AA:BB
eth.type: 2048
ip.src: 192.168.1.1
ip.dst: 192.168.1.2
ip.ttl: 64
udp.srcport: 12345
udp.dstport: 53
udp.length: 8

PacketId and FrameId convert implicitly from int, which is why 0 stands for both ids above; new PacketId(0) is the same call. The conversion checks the range like the constructor does: -1 is PacketId.Invalid, any other negative value or a value above ArrayIndexIdRange.MaxValue throws ArgumentOutOfRangeException. An id converts back to int for indexing, and id == 5 compares ids (the int side is converted and range-checked).

Packet ids of first parses must be dense on a stack: 0, 1, 2, … A gap returns ParseFailure.ParseIdGap. Parsing an id again is a re-parse and gives the same result.

Core Concepts

Namespaces: NetworkInspector.Core (stack, frame, packet, options), .Fields, .Ids, .Infos, .Settings, .Index, .ValueCaches, .Protocols (dissector interfaces); value types are in NetworkInspector.Values.

Common Tasks

Three Ways To Read Field Values

The snippets below are the smallest complete use of each. They run as tests (CoreApiSampleTests); stack and frame are the ones of the Quick Start, dstPort is stack.GetFieldId("udp.dstport")!.Value.

Field tree. Parse with the default options, then look fields up or walk the tree:

ParseOptions options = new();                                   // builds a field tree
Packet.TryParse(0, stack, frame, in options, out Packet? packet, out ParseFailure failure);

packet!.TryGetFieldValue(dstPort, out FieldValue value, materialize: true);   // first occurrence: 53

FieldLookupCookie cookie = FieldLookupCookie.Start;             // every occurrence (tunnels, VLAN tags)
while (packet.TryGetNextField(dstPort, ref cookie, out Field field, materialize: true))
{
    Console.WriteLine(field);                                   // the default display line: "Destination Port: 53"
}

field.ToString() (also $"{field}" and the allocation-free TryFormat, on MutField too) is the default display line: a custom text is the whole line (a dissector that wants the label writes it into the text); a field without a value shows its UI name; every other field shows the UI name, ": ", and the value's default string (its custom representation, else the default text of the data, which is also what field.Value.ToString() returns). A field id that the stack does not know (it has no FieldInfo) has no label, so its line is the value part alone. Every part stays readable, so a caller builds an own line from field.FieldInfo?.UiName (never empty: registration refuses an empty UI name), field.Value (Data, whose ToString is the default text of the data, and CustomRepresentation), and field.CustomText.

Value cache. Name the fields once, parse without a tree, read a column through a snapshot handle:

ValueCache cache = new(stack, [new ValueCacheFieldConfig(dstPort, ValueCaptureMode.FirstOccurrence)]);
ParseOptions options = new(FieldTreeMode.Skip, cache);
for (int i = 0; i < frames.Count; i++)
{
    Packet.TryParse(i, stack, frames[i], in options, out _, out _);
}

ValueCacheSeriesHandle<ulong> handle = cache.GetSeries<ulong>(dstPort).Handle;   // frozen Count, stable rows
foreach (ValueCacheRow<ulong> row in handle)
{
    Console.WriteLine(row.Value);
}

Field observer. Implement IFieldObserver, pair it with a watch, and parse without a tree:

sealed class PortRecorder(FieldId wanted) : IFieldObserver
{
    public void BeginPacket(PacketId id, Timestamp timestamp) { }

    public void OnField(in FieldVisit visit)
    {
        if (visit.FieldId == wanted)                            // ForExtraction also delivers sibling fields of the protocol
        {
            Console.WriteLine(visit.Value);                     // visit.CustomText: the text set at append
        }
    }

    public void EndPacket(PacketId id, bool complete) { }       // close or discard what you staged for this packet
}

ParseObserver[] observers = [new() { Observer = new PortRecorder(dstPort), Watch = FieldWatch.ForExtraction(stack, dstPort) }];
ParseOptions options = new(FieldTreeMode.Skip, observers: observers);
Packet.TryParse(0, stack, frame, in options, out _, out _);

Rules that apply to all three: ids of first parses are dense (0, 1, 2, …); a packet parsed with FieldTreeMode.Skip has no tree (HasFieldTree is false), so do not look fields up in it; an observer is called from the parse thread and is not thread-safe, and a cache has one writer (readers may read a snapshot handle concurrently). To feed several consumers from one parse, combine their needs in a ParsePlan.

Parse Many Frames With Recycling

Reuse one packet object for a stream of frames to avoid a heap allocation per packet:

ParseOptions options = new();
Packet? packet = null;
int nextId = 0;
foreach (Frame frame in frames)
{
    bool parsed = packet is null
        ? Packet.TryParse(nextId, stack, frame, in options, out packet, out ParseFailure failure)
        : Packet.TryParse(packet, nextId, stack, frame, in options, out failure);
    if (!parsed || packet is null)
    {
        continue;
    }

    nextId++;
    // Use the packet before the next iteration overwrites it.
}

Parse Without A Field Tree

When you only need a ValueCache, a PacketIndex, observers, or protocol side effects, skip the tree:

using NetworkInspector.Core.ValueCaches;

ParseOptions skip = new(FieldTreeMode.Skip, cache);
bool ok = Packet.TryParse(nextId, stack, frame, in skip, out Packet? throwaway, out ParseFailure failure);

Protocols still decode through the same code. The packet keeps no field bodies (HasFieldTree is false), so do not filter, store, or export it. The cache and the index record only on the first parse of an id; pass recordOnReplay: true to record on a re-parse. Default new ParseOptions() builds a tree.

Observe Fields During The Parse

A ParseObserver receives the fields of a parse while it runs, with or without a field tree. Pair an IFieldObserver with a FieldWatch (which fields) and, when you need display text, a DisplayText watch:

ParseObserver[] observers =
[
    new()
    {
        Observer = recorder,                                   // your IFieldObserver
        Watch = FieldWatch.ForExtraction(stack, srcPort),      // delivers srcPort even from deferred containers
        DisplayText = FieldWatch.Only(stack, srcPort),         // dissectors also build its custom text
    },
];
ParseOptions options = new(FieldTreeMode.Skip, observers: observers);

The observer sees BeginPacket(id, timestamp), OnField(in FieldVisit) for every accepted field, and then exactly one EndPacket(id, complete) per BeginPacket that ran, before the packet is sealed. complete is false when the observers were muted (an observer threw) or the parse aborted; a consumer that staged fields uses this call to close or discard them and to evaluate deferred custom text. FieldVisit.CustomText is a ref readonly to the text set when the field was appended: with a field tree it refers to the field's own slot. A deferred text builds its string once, however many copies of the struct exist, so keeping a copy (LazyString text = visit.CustomText;) is cheap and never builds the text twice. Read the text in EndPacket, not in OnField: a dissector may complete a deferred text after the append, and the first evaluation freezes the string for the tree and every copy. With a tree keep visit.Field and read Field.CustomText; without a tree keep the copied struct.

EndPacket is a required member of IFieldObserver: every observer implements it. An observer that keeps no state per packet leaves the body empty, as the examples above show.

• Why ForExtraction and not Only: Only expands a deferred container only when the container or a field of its index group is watched, so a field in a container outside that group is silently missing in a parse without tree. ForExtraction(stack, ids) accepts the ids plus every field of their owning protocols, so every container that can hold a listed field expands. Cost: more expansion, paid once per owning protocol.
• Why a separate DisplayText watch: formatting text costs work that filters and most observers do not need. With FieldWatch.None (the default) a parse without tree builds no custom text for the observer.
• Custom text is final at append. Observers see the text present at the append; changing it later (SetCustomText, AppendCustomText, ClearCustomText) is discouraged and throws InvalidOperationException after the packet is sealed. See the protocol guide §14.5.
• Observers run on every parse of a frame, including replays; the value cache and the index record only on the first parse unless recordOnReplay is set.

Combine Consumers Into One Parse Plan

Several consumers of one run (an exporter, a filter, a value cache) each state what they need as a ParseDemand; a ParsePlan takes the least parse work that serves all of them and parses each packet once:

ParseDemand[] demands =
[
    filter.ParseDemand,                                   // FilterObserver implements IParseDemandSource; list the filter first
    ParseDemand.Fields(new ParseObserver { Observer = exportObserver, Watch = exportWatch }),
];
if (!ParsePlan.TryCreate(stack, demands, out ParsePlan? plan, out ParsePlanError error))
{
    throw new InvalidOperationException(error.Message);   // e.g. two caches, a foreign cache, one observer in two demands
}

ParseOptions options = plan.Options;                      // Skip unless a demand needs the field tree
Console.WriteLine(plan);                                  // e.g. "fields only, no tree (2 fields, 2 observers)"

Why: the pipeline used to fix the parse work (the CLI always built a tree) and consumers adapted; an export of two values paid for the tree of every packet. Observers stay in demand order, so a filter listed first has its verdict before the exporter's packet is committed. One plan per stack and per parse thread; the observers are not thread-safe.

Collect Field Values Into Columns

Pass a ValueCache through ParseOptions to record selected fields during the first parse, then scan them column-wise with SIMD. Usage: VALUECACHE_GUIDE.md.

Read Captures And Export

Read frames with NetworkInspector.Sources (PCAP, PCAPNG, BLF, ASC) and write packets with NetworkInspector.Exporters (JSON, CSV, Text, PBF, Parquet, DuckDB). A packet exporter states what it needs as a ParseDemand; only a hierarchy export needs the field tree (see Combine Consumers Into One Parse Plan).

Read And Change Settings

SettingsManager holds typed Setting objects. A value has a text form for people and a JSON form for files; both validate against the type, range, and enumeration of the setting and never throw for a wrong value:

using NetworkInspector.Core.Settings;

using SettingsManager manager = new(folder);
manager.RegisterSetting(Setting.I64("net.retries", "Retries", "net", 3, min: 0, max: 10, activation: SettingActivation.SessionReset));

Setting retries = manager.GetSetting("net.retries")!;   // registered above
if (!SettingValueText.TryParse(retries, "25", out SettingValue value, out SettingProblem? problem)
    || !retries.TrySetPendingValue(value, out problem))
{
    Console.Error.WriteLine(problem);   // one line: the setting name and the allowed range
    return;
}

manager.ApplyChanges(["net.retries"]);   // pending → current for this key only
manager.SaveChanged(["net.retries"]);    // merges this key into its group file

• Two forms. SettingValueText (TryParse, Format) is the form of command lines and UIs; SettingValueJson (TryRead, Write) is the form of the files. Wrong input gives a SettingProblem with a SettingProblemKind (WrongType, BelowMinimum, AboveMaximum, NotFinite, UnknownEnum, InvalidPath, UnknownSetting) and one message that names the setting and the accepted values. Setting.TryValidate and Setting.TrySetPendingValue are the non-throwing counterparts of SetPendingValue.
• Lookup. TryGetSetting finds a setting by name; FindSimilar returns the closest names for a "did you mean".
• Activation. SettingActivation (Immediate, SessionReset, Restart) says when a host should use a changed value. Core applies every change at once; the host decides about restarts.
• Per-key apply. ApplyChanges(names) and ResetChanges(names) work on the given keys only; PendingChanged is raised for every key whose pending value changes, SettingsApplied for the applied ones.
• Sparse files. Save writes only values that differ from their default, one file per group, and writes the values of settings that are not registered in this run (Orphans, for example a plugin that is not loaded) back unchanged. Save writes the whole state of the manager: the key of every registered setting is set, or removed at its default, so a value that another process stored for a registered setting since Load is replaced (last writer wins). It deletes only group files that the manager saw at Load or wrote itself, never a file another process created after Load. SaveChanged(names) merges single keys into the files and keeps all other keys, also those another process wrote (names of several groups are written group by group; an error stops at that group and the groups before it stay written); Unset(name) removes a stored value so that the default applies again. Writers of several processes are serialized by the lock file .ni-settings.lock in the folder; a writer that waits more than two seconds gets a PersistenceSettingsException ("in use by another process; try again").
• Profiles. SettingsProfiles (List, Exists, TryCreate, TryRename, TryDelete) manages profile folders under a settings base; ProfileNames validates a name and StorageNames maps it to a folder name (names up to 120 characters are used as they are, longer ones become a shortened name with a hash and a .name file that keeps the full name). SettingsManagerFactory.ResolvePath(base, profile) gives the folder of a profile.

Write A Custom Dissector

Implement IProtocol.Parse and let the source generator write the registration code. Step by step: PROTOCOL_GUIDE.md.

Post-Parser Pipeline

Post-parsers are protocol-owned callbacks that run after the main protocol dispatch on every packet. Register them during stack construction with RegisterPostParser(protocolId, priority).

• Order: sorted once at build time, ascending by priority, then by registration order. No sorting while parsing.
• Lifecycle: they run after the dispatch tree, before packet.info is appended and before the packet is sealed. They receive the packet root as parent, so their fields are root-level siblings of the protocol fields.
• Index: in indexed parses (ParseOptions.Index) they run before PacketIndex.EndPacket; their presence records count like those of normal parsers.
• Errors: a ParseResult error or exception from a post-parser is recorded as packet.error. The remaining post-parsers still run. No error is silently discarded.

Limits And Thread-Safety Notes

• Build a stack once and reuse it. Stack construction is single-threaded.
• The first parse of each packet id is ordered and single-threaded; re-parses of already parsed ids may run concurrently. A sealed packet is safe for concurrent reads.
• Dense packet maps use ChunkedGrowOnlyStore<T>; packed effect logs use ChunkedAppendOnlyStore<T> (NetworkInspector.Core.Collections). Details: PROTOCOL_GUIDE.md.
• Treat frame bytes from captures as untrusted input; the dissectors validate lengths and report malformed data.
• NativeAOT and trimming are not supported.

Learn More

• Protocols · Sources · Exporters · Filter · Root overview
• Value cache guide · Protocol guide
• GitHub repository · Issue tracker

License

MIT License