Mostlylucid.LucidRAG.Storage.Core 2.7.5

Mostlylucid.Storage.Core

NuGet: Mostlylucid.LucidRAG.Storage.Core

Unified vector storage library for LucidRAG with support for InMemory, DuckDB, and Qdrant backends.

Overview

This library provides a single, unified interface (IVectorStore) for embedding storage and retrieval across all LucidRAG pipelines:

• DocumentPipeline - PDF, DOCX, Markdown text embeddings
• ImagePipeline - Image OCR, CLIP visual embeddings
• DataPipeline - Data profile embeddings for semantic search

Supported Backends

Quick Start

Installation

dotnet add package Mostlylucid.LucidRAG.Storage.Core

Tool/MCP Mode (No Persistence)

For one-shot analysis where you don't need to persist embeddings:

services.AddVectorStoreForToolMode();

Standalone Mode (DuckDB Persistence)

For standalone apps with embedded persistence:

services.AddVectorStoreForStandaloneMode(dataDirectory: "./data");

Production Mode (Qdrant)

For production deployments with dedicated vector database:

services.AddVectorStoreForProductionMode(qdrantHost: "localhost", qdrantPort: 6334);

Custom Configuration

services.AddVectorStore(options =>
{
    options.Backend = VectorStoreBackend.DuckDB;
    options.PersistVectors = true;
    options.ReuseExistingEmbeddings = true;

    options.DuckDB.DatabasePath = "./vectors.duckdb";
    options.DuckDB.EnableVSS = true;
    options.DuckDB.VectorDimension = 384;
});

Usage

Initialize Collection

var vectorStore = serviceProvider.GetRequiredService<IVectorStore>();

var schema = new VectorStoreSchema
{
    VectorDimension = 384,
    DistanceMetric = VectorDistance.Cosine,
    StoreText = true  // false for privacy-preserving mode
};

await vectorStore.InitializeAsync("documents", schema);

Insert Documents

var documents = new[]
{
    new VectorDocument
    {
        Id = "doc1:segment0",
        Embedding = new float[384], // your embedding vector
        ParentId = "doc1",
        ContentHash = ContentHasher.ComputeHash(text),
        Text = text,
        Metadata = new Dictionary<string, object>
        {
            ["type"] = "text",
            ["language"] = "en",
            ["confidence"] = 0.95
        }
    }
};

await vectorStore.UpsertDocumentsAsync("documents", documents);

Search

var query = new VectorSearchQuery
{
    QueryEmbedding = queryVector,
    TopK = 10,
    MinScore = 0.7,
    IncludeDocument = false,  // privacy-preserving - only return IDs
    Filters = new Dictionary<string, object>
    {
        ["language"] = "en"
    }
};

var results = await vectorStore.SearchAsync("documents", query);

foreach (var result in results)
{
    Console.WriteLine($"{result.Id}: {result.Score:F3}");
    Console.WriteLine($"  Metadata: {string.Join(", ", result.Metadata)}");
}

Content Hash-Based Caching

// Get documents by content hash (for deduplication)
var hashes = new[] { "abc123", "def456" };
var cached = await vectorStore.GetDocumentsByHashAsync("documents", hashes);

if (cached.TryGetValue("abc123", out var doc))
{
    Console.WriteLine("Reusing existing embedding!");
}

Incremental Updates

// Remove stale segments, keep new ones
var validHashes = new[] { "hash1", "hash2", "hash3" };
await vectorStore.RemoveStaleDocumentsAsync("documents", parentId: "doc1", validHashes);

Architecture

IMultiVectorStore : IVectorStore (unified interface)
├── InMemoryVectorStore     (ephemeral, fastest)
├── DuckDBVectorStore       (persistent, embedded, HNSW indexes)
└── QdrantVectorStore       (persistent, server-based, production)

Used by:
├── DocSummarizer.Core      (text embeddings)
├── ImageSummarizer.Core    (OCR + CLIP embeddings)
├── AudioSummarizer.Core    (voice embeddings via EmbeddingStorageWave)
├── DataSummarizer.Core     (data profile embeddings)
└── LucidRAG                (all of the above)

Configuration Reference

appsettings.json

{
  "VectorStore": {
    "Backend": "DuckDB",
    "CollectionName": "documents",
    "PersistVectors": true,
    "ReuseExistingEmbeddings": true,

    "DuckDB": {
      "DatabasePath": "./data/vectors.duckdb",
      "EnableVSS": true,
      "EnablePersistence": true,
      "VectorDimension": 384,
      "HNSW": {
        "M": 16,
        "EfConstruction": 200,
        "EfSearch": 100
      }
    },

    "Qdrant": {
      "Host": "localhost",
      "Port": 6334,
      "ApiKey": null,
      "VectorSize": 384,
      "UseHttps": false
    },

    "InMemory": {
      "MaxDocuments": 0,
      "Verbose": false
    }
  }
}

Backend Comparison

InMemory

Pros:

• Fastest (no disk I/O)
• No external dependencies
• Perfect for testing

Cons:

• Data lost on restart
• Limited by RAM
• No HNSW acceleration

Best for: MCP tools, one-shot CLI analysis, unit tests

DuckDB

Pros:

• File-based persistence
• No external server needed
• HNSW indexes (with VSS extension)
• Graceful fallback if VSS unavailable

Cons:

• Single-process (no distributed)
• HNSW persistence experimental

Best for: Standalone apps, development, embedded scenarios

Qdrant

Pros:

• Production-grade
• Distributed/multi-node
• Advanced filtering
• Multi-vector support

Cons:

• Requires external server
• More complex deployment

Best for: Production, multi-tenant, high-scale

Performance

DuckDB HNSW vs Brute-Force

Startup Time (Reindex on Restart)

Privacy-Preserving Mode

Set StoreText = false in schema to avoid storing plaintext:

var schema = new VectorStoreSchema
{
    VectorDimension = 384,
    StoreText = false  // Only store embeddings, not text
};

Search results will only return IDs and scores, not text content.

Multi-Vector Support

IMultiVectorStore extends IVectorStore with named vector support for multi-modal embeddings. All three backends ( InMemory, DuckDB, Qdrant) implement it.

public interface IMultiVectorStore : IVectorStore
{
    Task InitializeMultiVectorAsync(
        string collectionName,
        VectorStoreSchema primarySchema,
        IEnumerable<NamedVectorConfig> namedVectors,
        CancellationToken ct = default);

    Task UpsertMultiVectorDocumentsAsync(
        string collectionName,
        IEnumerable<MultiVectorDocument> documents,
        CancellationToken ct = default);

    Task<List<VectorSearchResult>> SearchByNamedVectorAsync(
        string collectionName,
        string vectorName,
        VectorSearchQuery query,
        CancellationToken ct = default);
}

Usage

var store = serviceProvider.GetRequiredService<IMultiVectorStore>();

// Initialize with primary + named vectors
await store.InitializeMultiVectorAsync("images", primarySchema, new[]
{
    new NamedVectorConfig { Name = "visual", Dimension = 512 },
    new NamedVectorConfig { Name = "color", Dimension = 128 },
});

// Upsert documents with named vectors
var doc = new MultiVectorDocument
{
    Id = "img1", Embedding = primaryEmbedding,
    NamedVectors = { ["visual"] = clipVector, ["color"] = colorVector }
};
await store.UpsertMultiVectorDocumentsAsync("images", [doc]);

// Search by named vector
var results = await store.SearchByNamedVectorAsync("images", "visual", query);

Backend Implementation Details

Enables separate embeddings for:

• Text (OCR from image)
• Visual (CLIP embedding)
• Color (color histogram)
• Motion (optical flow)
• Voice (ECAPA-TDNN speaker embedding)

Implementation Status

Phase 1: Foundation ✅ COMPLETE

• Create project structure
• Define IVectorStore interface
• Define VectorDocument, VectorSearchQuery, VectorSearchResult models
• Configuration classes with mode-specific factories
• DI extension methods

Phase 2: DuckDB Implementation ✅ COMPLETE

• Port DuckDBVectorStore from DataSummarizer
• VSS extension integration with graceful fallback
• In-memory cosine similarity fallback when VSS unavailable
• HNSW index configuration and creation
• Content hash-based caching for deduplication
• Summary caching support
• Schema migration for dimension changes
• All IVectorStore methods implemented

Key Features:

• ✅ VSS extension detection and loading
• ✅ Experimental persistence: SET hnsw_enable_experimental_persistence = true
• ✅ Dynamic vector dimensions (384 default, configurable)
• ✅ Vector literal syntax: [val1,val2,...]::FLOAT[dim]
• ✅ HNSW index with configurable M and ef_construction
• ✅ Dual search modes: VSS array_distance() or in-memory cosine
• ✅ Metadata filtering support
• ✅ Privacy-preserving mode (StoreText = false)

Phase 3: Remaining Implementations ✅ COMPLETE

• Port InMemoryVectorStore from DocSummarizer.Core
• Port QdrantVectorStore from DocSummarizer.Core
• Add IMultiVectorStore for multi-modal pipelines (image, audio)

InMemoryVectorStore Features:

• ✅ ConcurrentDictionary-based storage for thread-safety
• ✅ Cosine similarity search (brute-force, no index)
• ✅ LRU eviction with configurable MaxDocuments limit
• ✅ Content hash-based caching
• ✅ Summary caching
• ✅ Metadata filtering support
• ✅ Zero external dependencies

QdrantVectorStore Features:

• ✅ Production-grade persistent storage
• ✅ Qdrant gRPC client integration
• ✅ Batch upserts (100 points per batch)
• ✅ Filter-based search and deletion
• ✅ Content hash-based caching
• ✅ Summary caching with metadata
• ✅ Privacy-preserving mode support
• ✅ Configurable distance metrics (Cosine, Euclidean, Dot Product)
• ✅ HTTPS support

Phase 4: Migration

• Migrate DocSummarizer.Core
• Migrate DataSummarizer
• Migrate Mostlylucid.RAG
• Update LucidRAG web app

License

MIT - Part of the LucidRAG project