LLamaSharp 0.4.2-preview
See the version list below for details.
dotnet add package LLamaSharp --version 0.4.2-preview
NuGet\Install-Package LLamaSharp -Version 0.4.2-preview
<PackageReference Include="LLamaSharp" Version="0.4.2-preview" />
paket add LLamaSharp --version 0.4.2-preview
#r "nuget: LLamaSharp, 0.4.2-preview"
// Install LLamaSharp as a Cake Addin #addin nuget:?package=LLamaSharp&version=0.4.2-preview&prerelease // Install LLamaSharp as a Cake Tool #tool nuget:?package=LLamaSharp&version=0.4.2-preview&prerelease
LLamaSharp - .NET Binding for llama.cpp
The C#/.NET binding of llama.cpp. It provides APIs to inference the LLaMa Models and deploy it on local environment. It works on both Windows, Linux and MAC without requirment for compiling llama.cpp yourself. Its performance is close to llama.cpp.
Furthermore, it provides integrations with other projects such as BotSharp to provide higher-level applications and UI.
Documentation
Installation
Firstly, search LLamaSharp
in nuget package manager and install it.
PM> Install-Package LLamaSharp
Then, search and install one of the following backends:
LLamaSharp.Backend.Cpu
LLamaSharp.Backend.Cuda11
LLamaSharp.Backend.Cuda12
Here's the mapping of them and corresponding model samples provided by LLamaSharp
. If you're not sure which model is available for a version, please try our sample model.
LLamaSharp.Backend | LLamaSharp | Verified Model Resources | llama.cpp commit id |
---|---|---|---|
- | v0.2.0 | This version is not recommended to use. | - |
- | v0.2.1 | WizardLM, Vicuna (filenames with "old") | - |
v0.2.2 | v0.2.2, v0.2.3 | WizardLM, Vicuna (filenames without "old") | 63d2046 |
v0.3.0, v0.3.1 | v0.3.0, v0.4.0 | LLamaSharpSamples v0.3.0, WizardLM | 7e4ea5b |
v0.4.1-preview (cpu only) | v0.4.1-preview | Open llama 3b, Open Buddy | aacdbd4 |
v0.4.2-preview (cpu,cuda11) | v0.4.2-preview | Llama2 7b | - |
Many hands make light work. If you have found any other model resource that could work for a version, we'll appreciate it for opening an PR about it! 😊
We publish the backend with cpu, cuda11 and cuda12 because they are the most popular ones. If none of them matches, please compile the llama.cpp
from source and put the libllama
under your project's output path (guide).
FAQ
- GPU out of memory: Please try setting
n_gpu_layers
to a smaller number. - Unsupported model:
llama.cpp
is under quick development and often has break changes. Please check the release date of the model and find a suitable version of LLamaSharp to install, or use the model we provide on huggingface.
Usages
Model Inference and Chat Session
LLamaSharp provides two ways to run inference: LLamaExecutor
and ChatSession
. The chat session is a higher-level wrapping of the executor and the model. Here's a simple example to use chat session.
using LLama.Common;
using LLama;
string modelPath = "<Your model path>" // change it to your own model path
var prompt = "Transcript of a dialog, where the User interacts with an Assistant named Bob. Bob is helpful, kind, honest, good at writing, and never fails to answer the User's requests immediately and with precision.\r\n\r\nUser: Hello, Bob.\r\nBob: Hello. How may I help you today?\r\nUser: Please tell me the largest city in Europe.\r\nBob: Sure. The largest city in Europe is Moscow, the capital of Russia.\r\nUser:"; // use the "chat-with-bob" prompt here.
// Initialize a chat session
var ex = new InteractiveExecutor(new LLamaModel(new ModelParams(modelPath, contextSize: 1024, seed: 1337, gpuLayerCount: 5)));
ChatSession session = new ChatSession(ex);
// show the prompt
Console.WriteLine();
Console.Write(prompt);
// run the inference in a loop to chat with LLM
while (prompt != "stop")
{
foreach (var text in session.Chat(prompt, new InferenceParams() { Temperature = 0.6f, AntiPrompts = new List<string> { "User:" } }))
{
Console.Write(text);
}
prompt = Console.ReadLine();
}
// save the session
session.SaveSession("SavedSessionPath");
Quantization
The following example shows how to quantize the model. With LLamaSharp you needn't to compile c++ project and run scripts to quantize the model, instead, just run it in C#.
string srcFilename = "<Your source path>";
string dstFilename = "<Your destination path>";
string ftype = "q4_0";
if(Quantizer.Quantize(srcFileName, dstFilename, ftype))
{
Console.WriteLine("Quantization succeed!");
}
else
{
Console.WriteLine("Quantization failed!");
}
For more usages, please refer to Examples.
Web API
We provide the integration of ASP.NET core here. Since currently the API is not stable, please clone the repo and use it. In the future we'll publish it on NuGet.
Since we are in short of hands, if you're familiar with ASP.NET core, we'll appreciate it if you would like to help upgrading the Web API integration.
Demo
Roadmap
✅: completed. ⚠️: outdated for latest release but will be updated. 🔳: not completed
✅ LLaMa model inference
✅ Embeddings generation, tokenization and detokenization
✅ Chat session
✅ Quantization
✅ State saving and loading
⚠️ BotSharp Integration
⚠️ ASP.NET core Integration
⚠️ Semantic-kernel Integration
🔳 Fine-tune
🔳 Local document search
🔳 MAUI Integration
🔳 Follow up llama.cpp and improve performance
Assets
Some extra model resources could be found below:
- Qunatized models provided by LLamaSharp Authors
- eachadea/ggml-vicuna-13b-1.1
- TheBloke/wizardLM-7B-GGML
- Magnet: magnet:?xt=urn:btih:b8287ebfa04f879b048d4d4404108cf3e8014352&dn=LLaMA
The weights included in the magnet is exactly the weights from Facebook LLaMa.
The prompts could be found below:
Contributing
Any contribution is welcomed! Please read the contributing guide. You can do one of the followings to help us make LLamaSharp
better:
- Append a model link that is available for a version. (This is very important!)
- Star and share
LLamaSharp
to let others know it. - Add a feature or fix a BUG.
- Help to develop Web API and UI integration.
- Just start an issue about the problem you met!
Contact us
Join our chat on Discord.
Join QQ group
License
This project is licensed under the terms of the MIT license.
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 was computed. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 is compatible. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard2.0 is compatible. netstandard2.1 was computed. |
.NET Framework | net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen40 was computed. tizen60 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 2.0
- System.Linq.Async (>= 6.0.1)
- System.Text.Json (>= 6.0.0)
-
net6.0
- No dependencies.
-
net7.0
- No dependencies.
NuGet packages (10)
Showing the top 5 NuGet packages that depend on LLamaSharp:
Package | Downloads |
---|---|
Microsoft.KernelMemory.AI.LlamaSharp
Provide access to OpenAI LLM models in Kernel Memory to generate text |
|
LangChain.Providers.LLamaSharp
LLamaSharp Chat model provider. |
|
LLamaSharp.semantic-kernel
The integration of LLamaSharp and Microsoft semantic-kernel. |
|
LLamaSharp.kernel-memory
The integration of LLamaSharp and Microsoft kernel-memory. It could make it easy to support document search for LLamaSharp model inference. |
|
LangChain.Providers.Automatic1111
Automatic1111 Stable DIffusion model provider. |
GitHub repositories (4)
Showing the top 4 popular GitHub repositories that depend on LLamaSharp:
Repository | Stars |
---|---|
SciSharp/BotSharp
The AI Agent Framework in .NET
|
|
microsoft/kernel-memory
RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.
|
|
CodeMazeBlog/CodeMazeGuides
The main repository for all the Code Maze guides
|
|
jxq1997216/AITranslator
使用大语言模型来翻译MTool导出的待翻译文件的图像化UI软件
|
Version | Downloads | Last updated |
---|---|---|
0.18.0 | 2,925 | 10/19/2024 |
0.17.0 | 673 | 10/13/2024 |
0.16.0 | 40,417 | 9/1/2024 |
0.15.0 | 16,635 | 8/3/2024 |
0.14.0 | 2,991 | 7/16/2024 |
0.13.0 | 47,602 | 6/4/2024 |
0.12.0 | 77,958 | 5/12/2024 |
0.11.2 | 14,423 | 4/6/2024 |
0.11.1 | 834 | 3/31/2024 |
0.10.0 | 6,189 | 2/15/2024 |
0.9.1 | 9,854 | 1/6/2024 |
0.9.0 | 543 | 1/6/2024 |
0.8.1 | 21,150 | 11/28/2023 |
0.8.0 | 19,676 | 11/12/2023 |
0.7.0 | 1,843 | 10/31/2023 |
0.6.0 | 2,366 | 10/24/2023 |
0.5.1 | 6,890 | 9/5/2023 |
0.4.2-preview | 1,942 | 8/6/2023 |
0.4.1-preview | 1,244 | 6/21/2023 |
0.4.0 | 11,343 | 6/19/2023 |
0.3.0 | 10,674 | 5/22/2023 |
0.2.3 | 711 | 5/17/2023 |
0.2.2 | 636 | 5/17/2023 |
0.2.1 | 673 | 5/12/2023 |
0.2.0 | 839 | 5/12/2023 |
LLamaSharp 0.4.1 followed up the master branch of llama.cpp. (commit id: aacdbd4)