LuceneSharp.Spatial 26.7.2622

LuceneSharp — A from-scratch port of Apache Lucene to modern .NET 10

Status: functional. All Tier 1–3 modules are ported and green. The solution builds clean on the .NET 10 SDK (warnings-as-errors, public API fully XML-documented) and ~8,600 tests pass across 34 test projects, with two runnable demos (full-text search and HNSW vector search). What remains is tracked in TODO.md: mostly proof/infra (a .NET CI pipeline, Java↔C# format-compat fixtures needing a JVM, a NativeAOT publish smoke test), the Mosaik integration (separate repo), and a handful of niche/optional items (Kuromoji n-best, uniformsplit/sharedterms, PerfTask/MoreLikeThisTask). Items blocked by this environment (ICU/Thai/OpenNLP native models, the JVM for cross-format fixtures) are marked [!].

LuceneSharp is an independent port of Apache Lucene (Java) to .NET 10 / C# 14, written from scratch with idiomatic, modern .NET in mind.

It is not a continuation of the venerable Lucene.Net project. We do not attempt a line-by-line translation of the Java sources or a machine port from Java byte code. Instead, we treat the Java reference implementation as a specification — the file format, the algorithms, the public API shape — and reimplement it in C# using the platform's strengths:

• Span<T> / Memory<T> / ReadOnlySequence<T> end-to-end (no allocations on the read path; no byte[] shuffle between layers).
• System.Numerics.Tensors and the cross-platform SIMD APIs in System.Runtime.Intrinsics and System.Numerics.Vector<T> for vector math, distance functions, posting list decoding, and codec hot loops.
• MemoryMappedFile, RandomAccess and direct SafeFileHandle access for Directory implementations — the equivalent of Java's MemorySegment/Panama APIs.
• async/await and ValueTask where it actually helps (async commit/flush, replicator, monitor); blocking APIs for hot search paths.
• Source generators where they replace runtime reflection (codec SPI discovery, attribute factories, expression compilation).
• Modern C# language features — readonly record struct, primary constructors, collection expressions, generic math (INumber<T>), required members, file-scoped types — without keeping a "looks like Java" facade.

The goal is format-compatibility, not source-compatibility: indexes written by Apache Lucene must be readable by LuceneSharp, and vice versa.

Why a new port?

Lucene.Net is an excellent project, but its design priorities (line-by-line fidelity to the Java sources, support for very old TFMs, exposed-as-Java public surface) make it hard to take advantage of features that have shipped in .NET since 2020: hardware-accelerated tensor math, Span<T> everywhere, file I/O without Stream shims, INumber<T> generic math, source generators in place of reflection, modern async, AOT compatibility, NativeAOT trimming, and a much shorter list of unsafe blocks because most of the same work can now be done with MemoryMarshal / Unsafe.As.

Modern Apache Lucene (10.x) leans heavily on Java's Panama Vector API and MemorySegment; the natural .NET equivalents (System.Numerics.Tensors, Vector256<T>, MemoryMappedFile+Span<byte>) deserve a port designed around them from the start.

Scope

The port aims for functional parity with Apache Lucene 10.x — same on-disk index format, same default codec (Lucene104), same query semantics, same scoring (BM25 default), same HNSW vector search behavior.

Public API names are anglicized into .NET conventions (PascalCase, properties instead of getX()/setX(), IDisposable instead of Closeable, iterators that return IEnumerable<T>/IAsyncEnumerable<T> where appropriate). See docs/api-conventions.md (TODO) for the full mapping.

Priority order (driven by Mosaik's actual Lucene usage)

1. Tier 1 (Critical — full-text indexing/search): Util core types, Store (Directory abstraction, FSDirectory, ByteBuffersDirectory, MMapDirectory, RAMDirectory-equivalent), Index (IndexWriter / Reader, segment management), Document, Search (IndexSearcher, core Query types, BM25Similarity, Collectors), the default Lucene104 codec.
2. Tier 1 (Critical — vector search): Util.Hnsw, Util.Quantization, Codecs.Hnsw, KNN search infrastructure (KnnFloatVectorField, KnnByteVectorField, KnnFloatVectorQuery, etc.).
3. Tier 2 (Important): Analysis.Standard + Analysis.Common minimum (StandardAnalyzer, NGram, Whitespace, LowerCase, KeywordAnalyzer); QueryParser classic + flexible; Queries (custom function/score queries); Highlighter; Facet; Grouping; Memory (single-doc MemoryIndex); Misc; Sandbox.
4. Tier 3 (Nice to have): Per-language analyzers (European first — en, de, fr, es, it, pt, nl, pl, sv, da, no, fi); Suggest; Spatial/Spatial3D; Join; Monitor; Classification; Replicator; Expressions.
5. Tier 4 (Deferred): CJK / RTL / Indic analyzers; ICU integration; Kuromoji (Japanese), Nori (Korean), SmartCN (Chinese), OpenNLP, Morfologik, Stempel, Phonetic.

Format scope: LuceneSharp targets only the latest on-disk format (currently Lucene104) and anything that supersedes it in the future. Backward codecs and legacy-format readers are explicitly out of scope — indexes written by older Apache Lucene / Lucene.Net releases must be reindexed with upstream tooling before they can be opened by LuceneSharp. See docs/format-compatibility.md.

See TODO.md for the complete, project-by-project breakdown.

Repository layout

dotnet/
  src/
    Lucene.Core/                   Core indexing, search, store, util, codecs
    Lucene.Analysis.Common/        StandardAnalyzer, language analyzers, filters
    Lucene.Codecs/                 Optional/experimental codecs (no legacy readers)
    Lucene.Queries/                Extra queries (function, payload, custom score)
    Lucene.QueryParser/            Classic and flexible query parsers
    Lucene.Highlighter/            Result highlighting
    Lucene.Facet/                  Faceted search
    Lucene.Grouping/               Result grouping
    Lucene.Join/                   Block-join and query-time join
    Lucene.Suggest/                Auto-suggest, spellcheck
    Lucene.Memory/                 Single-doc in-memory index
    Lucene.Misc/                   Tools (CheckIndex, IndexSplitter, ...)
    Lucene.Monitor/                Reverse-search (Luwak-style)
    Lucene.Replicator/             Index replication
    Lucene.Spatial/                Geospatial search (2D)
    Lucene.Spatial3D/              3D spatial geometry
    Lucene.Sandbox/                Experimental contributions
    Lucene.Expressions/            JavaScript-like expression compiler
    Lucene.Classification/         Bayes / k-NN classifiers
    Lucene.Demo/                   Example programs
    Lucene.TestFramework/          Shared test utilities (random docs, mock dirs)
  test/
    Lucene.Core.Tests/             Mirrors src/LuceneSharp.Core
    Lucene.<Module>.Tests/         One test project per src module
    Lucene.Integration.Tests/      Cross-module integration + format-compat
    Lucene.Benchmarks/             BenchmarkDotNet harness
  samples/
    Lucene.Demo/                   End-to-end indexing+search demo
    Lucene.VectorSearch.Demo/      HNSW vector search demo
  build/                           Directory.Build.props/.targets, NuGet config
  docs/                            Design notes, API conventions, migration
  TODO.md                          Full porting plan
  CLAUDE.md                        Guidance for AI agents working on the port

Each source folder mirrors the corresponding package in lucene-sharp/lucene/<module>/src/java/org/apache/lucene/.... Tests under test/LuceneSharp.<Module>.Tests/... mirror the Java tests under lucene-sharp/lucene/<module>/src/test/....

Building

Requires the .NET 10 SDK.

cd dotnet
dotnet build Lucene.slnx -c Release
dotnet test  Lucene.slnx -c Release

The build is warnings-as-errors and the public API is fully XML-documented (missing docs on a public member fail the build). To run a single module's tests, point dotnet test at that project, e.g. dotnet test test/LuceneSharp.Core.Tests/LuceneSharp.Core.Tests.csproj -c Release.

Try the demos

# Full-text indexing + classic-parser search (BM25-ranked hits):
dotnet run --project samples/LuceneSharp.Demo -c Release -- --query "lucene"

# HNSW vector search (cosine k-NN):
dotnet run --project samples/LuceneSharp.VectorSearch.Demo -c Release

Quick start (indexing + search)

using Lucene.Analysis.Standard;
using Lucene.Document;
using Lucene.Index;
using Lucene.Search;
using Lucene.Store;

using var dir = new ByteBuffersDirectory();
var analyzer = new StandardAnalyzer();

using (var writer = new IndexWriter(dir, new IndexWriterConfig(analyzer)))
{
    var doc = new Document();
    doc.Add(new TextField("body", "LuceneSharp is a from-scratch .NET port", Field.Store.Yes));
    writer.AddDocument(doc);                 // or: await writer.AddDocumentAsync(doc);
    writer.Commit();                         // or: await writer.CommitAsync();
}

using var reader = DirectoryReader.Open(dir);
var searcher = new IndexSearcher(reader);
TopDocs hits = searcher.Search(new TermQuery(new Term("body", "port")), 10);

Format compatibility

The port targets binary read/write compatibility with the current default Apache Lucene codec (currently Lucene104Codec). Older formats are not supported in either direction — reindex with upstream Apache Lucene first. A round-trip test suite under test/LuceneSharp.Integration.Tests/FormatCompat/ writes indexes with the Java reference implementation (via a small process harness against pre-generated fixtures) and reads them back with the .NET port, and vice versa. See docs/format-compatibility.md for the full policy.

License

Apache License 2.0 — same as Apache Lucene. The Java sources in lucene/ remain under the ASF copyright; new C# sources under dotnet/ carry the standard Apache 2.0 header.

Contributing

See CONTRIBUTING.md at the repo root for the Apache Lucene contribution guide (issues / mailing lists), and dotnet/CLAUDE.md / dotnet/docs/ for guidance specific to the C# port.