Microsoft.KernelMemory.MemoryDb.Postgres 0.94.241201.1

Prefix Reserved
This package has a SemVer 2.0.0 package version: 0.94.241201.1+77fd7be.
There is a newer version of this package available.
See the version list below for details.
dotnet add package Microsoft.KernelMemory.MemoryDb.Postgres --version 0.94.241201.1                
NuGet\Install-Package Microsoft.KernelMemory.MemoryDb.Postgres -Version 0.94.241201.1                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Microsoft.KernelMemory.MemoryDb.Postgres" Version="0.94.241201.1" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Microsoft.KernelMemory.MemoryDb.Postgres --version 0.94.241201.1                
#r "nuget: Microsoft.KernelMemory.MemoryDb.Postgres, 0.94.241201.1"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Microsoft.KernelMemory.MemoryDb.Postgres as a Cake Addin
#addin nuget:?package=Microsoft.KernelMemory.MemoryDb.Postgres&version=0.94.241201.1

// Install Microsoft.KernelMemory.MemoryDb.Postgres as a Cake Tool
#tool nuget:?package=Microsoft.KernelMemory.MemoryDb.Postgres&version=0.94.241201.1                

Kernel Memory with Postgres + pgvector

Nuget package Discord

This project contains the Postgres adapter allowing to use Kernel Memory with Postgres+pgvector.

[!IMPORTANT] Your Postgres instance must support vectors. You can run this SQL to see the list of extensions installed and enabled:

SELECT * FROM pg_extension

To enable the extension this should suffice:

CREATE EXTENSION vector

For more information, check:

To use Postgres with Kernel Memory:

  1. Have a PostgreSQL instance ready, e.g. checkout Azure Database for PostgreSQL

  2. Verify your Postgres instance supports vectors, e.g. run SELECT * FROM pg_extension

  3. Add Postgres connection string to appsettings.json (or appsettings.development.json), for example:

    {
      "KernelMemory": {
        "Services": {
          "Postgres": {
            "ConnectionString": "Host=localhost;Port=5432;Username=myuser;Password=mypassword;Database=mydatabase"
          }
        }
      }
    }
    
  4. Configure KM builder to store memories in Postgres, and to persist documents, for example:

    // using Microsoft.KernelMemory;
    // using Microsoft.KernelMemory.Postgres;
    // using Microsoft.Extensions.Configuration;
    
    var postgresConfig = new PostgresConfig();
    
    new ConfigurationBuilder()
        .AddJsonFile("appsettings.json")
        .AddJsonFile("appsettings.development.json", optional: true)
        .AddJsonFile("appsettings.Development.json", optional: true)
        .Build()
        .BindSection("KernelMemory:Services:Postgres", postgresConfig);
    
    var memory = new KernelMemoryBuilder()
        .WithPostgresMemoryDb(postgresConfig)
        .WithSimpleFileStorage(SimpleFileStorageConfig.Persistent)
        .WithAzureOpenAITextGeneration(azureOpenAIConfig)
        .WithAzureOpenAITextEmbeddingGeneration(azureOpenAIConfig)
        .Build();
    

Neighbor search indexes, quality and performance

The connector does not create IVFFlat or HNSW indexes on Postgres tables, and uses exact nearest neighbor search. HNSW (Hierarchical Navigable Small World) has been introduced in pgvector 0.5.0 and is not available in some Postgres instances.

Depending on your scenario you might want to create these indexes manually, considering precision and performance trade-offs, or you can customize the SQL used to create tables via configuration.

[!NOTE] An IVFFlat index divides vectors into lists, and then searches a subset of those lists that are closest to the query vector. It has faster build times and uses less memory than HNSW, but has lower query performance (in terms of speed-recall tradeoff).

SQL to add IVFFlat: CREATE INDEX ON %%table_name%% USING ivfflat (embedding vector_cosine_ops) WITH (lists = 1000);

[!NOTE] An HNSW index creates a multilayer graph. It has slower build times and uses more memory than IVFFlat, but has better query performance (in terms of speed-recall tradeoff). There’s no training step like IVFFlat, so the index can be created without any data in the table.

SQL to add HNSW: CREATE INDEX ON %%table_name%% USING hnsw (embedding vector_cosine_ops);

See https://github.com/pgvector/pgvector for more information.

Memory Indexes and Postgres tables

The Postgres memory connector will create "memory indexes" automatically, one DB table for each memory index.

Table names have a configurable prefix, used to filter out other tables that might be present. The prefix is mandatory, cannot be empty, we suggest using the default km- prefix. Note that the Postgres connector automatically converts _ underscores to - dashes to have a consistent behavior with other storage types supported by Kernel Memory.

Overall we recommend not mixing external tables in the same DB used for Kernel Memory.

Column names and table schema

The connector uses a default schema with predefined columns and indexes.

You can change the field names, and if you need to add additional columns or indexes, you can also customize the CREATE TABLE SQL statement. You can use this approach, for example, to use IVFFlat or HNSW.

See PostgresConfig class for more details.

Here's an example where PostgresConfig is stored in appsettings.json and the table schema is customized, with custom names and additional fields.

The SQL statement requires two special placeholders:

  • %%table_name%%: replaced at runtime with the table name
  • %%vector_size%%: replaced at runtime with the embedding vectors size

There's a third optional placeholder we recommend using, to better handle concurrency, e.g. in combination with pg_advisory_xact_lock (exclusive transaction level advisory locks):

  • %%lock_id%%: replaced at runtime with a number

Also:

  • TableNamePrefix is mandatory string added to all KM tables
  • Columns is a required map describing where KM will store its data. If you have additional columns you don't need to list them here, only in SQL statement.
  • CreateTableSql is your optional custom SQL statement used to create tables. The column names must match those used in Columns.
{
  "KernelMemory": {
    "Services": {
      "Postgres": {

        "TableNamePrefix": "memory_",

        "Columns": {
          "id":        "_pk",
          "embedding": "embedding",
          "tags":      "labels",
          "content":   "chunk",
          "payload":   "extras"
        },

        "CreateTableSql": [
          "BEGIN;                                                                      ",
          "SELECT pg_advisory_xact_lock(%%lock_id%%);                                  ",
          "CREATE TABLE IF NOT EXISTS %%table_name%% (                                 ",
          "  _pk         TEXT NOT NULL PRIMARY KEY,                                    ",
          "  embedding   vector(%%vector_size%%),                                      ",
          "  labels      TEXT[] DEFAULT '{}'::TEXT[] NOT NULL,                         ",
          "  chunk       TEXT DEFAULT '' NOT NULL,                                     ",
          "  extras      JSONB DEFAULT '{}'::JSONB NOT NULL,                           ",
          "  my_field1   TEXT DEFAULT '',                                              ",
          "  _update     TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP            ",
          ");                                                                          ",
          "CREATE INDEX ON %%table_name%% USING GIN(labels);                           ",
          "CREATE INDEX ON %%table_name%% USING ivfflat (embedding vector_cosine_ops); ",
          "COMMIT;                                                                     "
        ]

      }
    }
  }
}
Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on Microsoft.KernelMemory.MemoryDb.Postgres:

Package Downloads
Microsoft.KernelMemory

The package contains all the core logic and extensions of Kernel Memory, to index and query any data and documents, using LLM and natural language, tracking sources and showing citations.

Aip.Km.KernelMemoryExtension

Package Description

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.95.241216.2 377 12/17/2024
0.95.241216.1 67 12/16/2024
0.94.241201.1 1,576 12/1/2024
0.93.241118.1 5,817 11/19/2024
0.92.241112.1 877 11/12/2024
0.91.241101.1 4,076 11/1/2024
0.91.241031.1 310 10/31/2024
0.90.241021.1 4,001 10/22/2024
0.90.241020.3 142 10/20/2024
0.80.241017.2 2,595 10/17/2024
0.79.241014.2 2,294 10/14/2024
0.79.241014.1 195 10/14/2024
0.78.241007.1 3,764 10/8/2024
0.78.241005.1 536 10/6/2024
0.77.241004.1 375 10/5/2024
0.76.240930.3 6,009 9/30/2024
0.75.240924.1 9,456 9/24/2024
0.74.240919.1 3,328 9/19/2024
0.73.240906.1 23,465 9/7/2024
0.72.240904.1 3,010 9/5/2024
0.71.240820.1 13,624 8/21/2024
0.70.240803.1 23,301 8/3/2024
0.69.240727.1 9,342 7/27/2024
0.68.240722.1 3,106 7/22/2024
0.68.240716.1 2,135 7/16/2024
0.67.240712.1 1,849 7/12/2024
0.66.240709.1 5,812 7/9/2024
0.65.240620.1 32,951 6/21/2024
0.64.240619.1 789 6/20/2024
0.63.240618.1 2,957 6/18/2024
0.62.240605.1 23,540 6/5/2024
0.62.240604.1 585 6/4/2024
0.61.240524.1 10,236 5/24/2024
0.61.240519.2 10,143 5/19/2024
0.60.240517.1 217 5/18/2024
0.51.240513.2 7,333 5/13/2024
0.50.240504.7 5,572 5/4/2024
0.40.240501.1 225 5/1/2024 0.40.240501.1 is deprecated because it is no longer maintained.
0.39.240427.1 1,392 4/28/2024 0.39.240427.1 is deprecated because it is no longer maintained.
0.38.240425.1 206 4/25/2024 0.38.240425.1 is deprecated because it is no longer maintained.
0.38.240423.1 537 4/24/2024 0.38.240423.1 is deprecated because it is no longer maintained.
0.37.240420.2 601 4/21/2024 0.37.240420.2 is deprecated because it is no longer maintained.
0.36.240416.1 1,637 4/16/2024 0.36.240416.1 is deprecated because it is no longer maintained.
0.36.240415.2 181 4/16/2024 0.36.240415.2 is deprecated because it is no longer maintained.
0.36.240415.1 117 4/15/2024 0.36.240415.1 is deprecated because it is no longer maintained.
0.35.240412.2 146 4/12/2024 0.35.240412.2 is deprecated because it is no longer maintained.
0.35.240321.1 2,280 3/21/2024 0.35.240321.1 is deprecated because it is no longer maintained.
0.35.240318.1 19,372 3/18/2024 0.35.240318.1 is deprecated because it is no longer maintained.
0.34.240313.1 631 3/13/2024 0.34.240313.1 is deprecated because it is no longer maintained.
0.33.240312.1 176 3/12/2024 0.33.240312.1 is deprecated because it is no longer maintained.
0.32.240308.1 579 3/8/2024 0.32.240308.1 is deprecated because it is no longer maintained.
0.32.240307.3 158 3/7/2024 0.32.240307.3 is deprecated because it is no longer maintained.
0.32.240307.2 114 3/7/2024 0.32.240307.2 is deprecated because it is no longer maintained.
0.30.240227.1 13,380 2/28/2024 0.30.240227.1 is deprecated because it is no longer maintained.
0.29.240219.2 3,381 2/20/2024 0.29.240219.2 is deprecated because it is no longer maintained.
0.28.240212.1 501 2/13/2024 0.28.240212.1 is deprecated because it is no longer maintained.
0.27.240207.1 235 2/7/2024 0.27.240207.1 is deprecated because it is no longer maintained.
0.27.240205.2 244 2/6/2024 0.27.240205.2 is deprecated because it is no longer maintained.
0.27.240205.1 233 2/5/2024 0.27.240205.1 is deprecated because it is no longer maintained.
0.26.240121.1 3,942 1/22/2024 0.26.240121.1 is deprecated because it is no longer maintained.
0.26.240116.2 601 1/16/2024 0.26.240116.2 is deprecated because it is no longer maintained.
0.26.240115.4 335 1/16/2024 0.26.240115.4 is deprecated because it is no longer maintained.
0.26.240104.1 1,328 1/5/2024 0.26.240104.1 is deprecated because it is no longer maintained.
0.25.240103.1 178 1/4/2024 0.25.240103.1 is deprecated because it is no longer maintained.
0.24.231228.5 568 12/29/2023 0.24.231228.5 is deprecated because it is no longer maintained.
0.24.231228.4 625 12/29/2023 0.24.231228.4 is deprecated because it is no longer maintained.