NMaier.PlaneDB
0.11.1
dotnet add package NMaier.PlaneDB --version 0.11.1
NuGet\Install-Package NMaier.PlaneDB -Version 0.11.1
<PackageReference Include="NMaier.PlaneDB" Version="0.11.1" />
paket add NMaier.PlaneDB --version 0.11.1
#r "nuget: NMaier.PlaneDB, 0.11.1"
// Install NMaier.PlaneDB as a Cake Addin #addin nuget:?package=NMaier.PlaneDB&version=0.11.1 // Install NMaier.PlaneDB as a Cake Tool #tool nuget:?package=NMaier.PlaneDB&version=0.11.1
PlaneDB
Kinda like LevelDB, but in C#
What?
PlaneDB is a key-value store or database written in pure C#. It uses a log structured merge tree approach, with a memory database with journal for recently changed keys, sorted string tables (sstable) in multiple levels for on disk storage, and a block based approach, allowing features such as on-disk data compression or encryption.
Aside from the key-value PlaneDB implementations (PlaneDB
, TypedPlaneDB<TKey, TValue>
, StringPlaneDB
), there are PlaneSet implementations (PlaneSet
, TypedPlaneSet<T>
, StringPlaneSet
) using the same underlying architecture, but optimized for storing sets.
Features
- General byte-array based key-value store (
PlaneDB
), and set (PlaneSet
) - Typed key-value/set store using serializers (
TypedPlaneDB
,TypedPlaneSet
) including a string stores (StringPlaneDB
,StringPlaneSet
) - Configurable on-disk compression, encryption. Uses my
BlockStream
library, which comes with lz4 compression and a ChaCha20-Poly1305 cipher/mac implementation. - Customizable byte-array comparer.
- Thread-safety (and Task-safety) by default.
- Crash resilience using read-only/write-once tables and a journal for the most recent changes. By default the journal is flushed either once every 2 seconds, or whenever a certain amount of write operations were performed to conserve disk writes, meaning the last few operations may be lost upon unclean shutdown, but this can be configured to flush upon each write
.MakeFullySync()/.FLushJournalAfterNumerOfWrite(...)
. - Automatic bloom filters to avoid useless reads, using
xxHash
hashes. - API interface implementing
IDictionary<>
and variousConcurrentDictionary
methods. - API interface implementing
ISet<>
and various additional methods. - Multiple stores (table spaces) in a single directory.
- Data (sstable) files, once written, are read-only. Additional data is always stored in new files (after first collected in a memory table), which maybe be merged together into yet new files (higher levels).
Example
var location = new DirectoryInfo(".");
var options = new PlaneDBOptions().EnableCompression().MakeFullySync();
using (var db = new StringPlaneDB(location, FileMode.CreateNew, options) {
db["abc"] = "def"; // Direct set
db.AddOrUpdate("abc", "ghi", (_, __) => "jkl"); // Will call the update function
db.TryAdd("abc", "mno"); // Ignored
}
using (var db = new StringPlaneDB(location, FileMode.OpenOrCreate, options) {
Console.WriteLine($"abc={db["abc"]}"); // abc=jkl
}
Word of advise: Do not excessively use .Count
/.Keys
/.Values
, as these properties need to perform a full iteration over the entire data set.
Performance
Hmmm... good enough? In my unscientific synthetic benchmarks the performance is comparable to that of LevelDB, with wallclock times between 0.4x - 2.0x compared to RocksDB(Sharp), depending on workload and configuration.
Limitations
- No support for column families or such (yet)
- Keys and values have to fit into your memory. Not at the same time, but whenever they are read. While large keys values up to 2GB each are supported in theory, in practise you're bound by memory and also this has been untested.
File Structure
LOCK
- To ensure exclusivity. Always empty.MANIFEST
- Stores information about what sstable files belong to what level and the generation counterJOURNAL
- Recovery information in case of unclean shutdowns.###
- readonly sstable files storing the actual key and value data, as well as pre-computed bloom filters.
Status
Beta-grade software. I have been dogfooding the code for a while in various internal applications of mine (none business-critical), with millions of records, along with the test suite.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 is compatible. net5.0-windows was computed. net6.0 was computed. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.1 is compatible. |
.NET Framework | net48 is compatible. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETFramework 4.8
- NMaier.BlockStream (>= 1.7.4)
- NMaier.Serializers (>= 1.2.0)
-
.NETStandard 2.1
- NMaier.BlockStream (>= 1.7.4)
- NMaier.Serializers (>= 1.2.0)
-
net5.0
- NMaier.BlockStream (>= 1.7.4)
- NMaier.Serializers (>= 1.2.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.