Microsoft.ML.OnnxRuntimeGenAI.Foundry 0.19.0

About

Run Llama, Phi (language and multi modal!), DeepSeek, Gemma, Mistral with ONNX Runtime.

This API gives you an easy, flexible and performant way of running LLMs on device using .NET/C#.

It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, and KV cache management.

Key Features

• Language, vision, and audio pre and post processing
• Inference using ONNX Runtime
• Generation tuning with greedy, beam search and random sampling
• KV cache management to optimize performance
• Multi target execution (CPU, GPU, with NPU coming!)

Sample

// See https://aka.ms/new-console-template for more information
using Microsoft.ML.OnnxRuntimeGenAI;

using OgaHandle ogaHandle = new OgaHandle();

// Specify the location of your downloaded model.
// Many models are published on HuggingFace e.g. 
// https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx
string modelPath = "..."
Console.WriteLine("Model path: " + modelPath);

using Model model = new Model(modelPath);
using Tokenizer tokenizer = new Tokenizer(model);
using var tokenizerStream = tokenizer.CreateStream();

// Set your prompt here
string prompt = "public static bool IsPrime(int number)";
var sequences = tokenizer.Encode($"<|user|>{prompt}<|end|><|assistant|>");

using GeneratorParams generatorParams = new GeneratorParams(model);
generatorParams.SetSearchOption("max_length", 512);

using var generator = new Generator(model, generatorParams);
generator.AppendTokenSequences(sequences);
.
while (!generator.IsDone())
{
    generator.GenerateNextToken();
    Console.Write(tokenizerStream.Decode(generator.GetSequence(0)[^1]));
}

Generates the following output:

Here's a complete implementation of the `IsPrime` function in C# that checks if a given number is prime. The function includes basic input validation and comments for clarity.

using System;

namespace PrimeChecker
{
    public class PrimeChecker
    {
        /// <summary>
        /// Checks if the given number is prime.
        /// </summary>
        /// <param name="number">The number to check.</param>
        /// <returns>true if the number is prime; otherwise, false.</returns>
        public static bool IsPrime(int number)
        {
            // Input validation
            if (number < 2)
            {
                return false;
            }

            // 2 is the only even prime number
            if (number == 2)
            {
                return true;
            }

            // Exclude even numbers greater than 2
            if (number % 2 == 0)
            {
                return false;
            }

            // Check for factors up to the square root of the number
            int limit = (int)Math.Floor(Math.Sqrt(number));
            for (int i = 3; i <= limit; i += 2)
            {
                if (number % i == 0)
                {
                    return false;
                }
            }

            return true;
        }

        static void Main(string[] args)
        {
            int number = 29;
            bool isPrime = PrimeChecker.IsPrime(number);

            Console.WriteLine($"Is {number} prime? {isPrime}");
        }
    }
}

This implementation checks if a number is prime by iterating only up to the square root of the number, which is an optimization over checking all numbers up to the number itself. It also excludes even numbers greater than 2, as they cannot be prime.

Source code repository

ONNX Runtime is an open source project. See:

• ONNX Runtime
• ONNX Runtime GenAI

Documentation

See ONNX Runtime GenAI Documentation

ONNX Runtime dependency

GenAI NuGet packages do not bundle ONNX Runtime binaries. ONNX Runtime is supplied by separate runtime packages. The WinML NuGet depends on Microsoft.Windows.AI.MachineLearning for its runtime.

Linux ARM64 CUDA

The CUDA NuGet contains the Linux ARM64 GenAI core and CUDA add-on under runtimes/linux-arm64/native, but does not bundle ONNX Runtime libraries. Linux ARM64 consumers must provide a compatible ONNX Runtime GPU 1.30.0 native installation separately, along with a CUDA 13.1 runtime and a compatible NVIDIA driver. Other RIDs continue to obtain ONNX Runtime from the NuGet dependency. The .Foundry package carries the Linux ARM64 CUDA GenAI core, but deliberately excludes the CUDA shim and ONNX Runtime libraries, as it does for Linux x64. Its Linux ARM64 consumer must provide a compatible ONNX Runtime GPU 1.30.0 native installation separately.

0.19.0 migration note

The Linux ARM64 CUDA NuGet no longer includes ONNX Runtime binaries. Applications must obtain a compatible ONNX Runtime GPU 1.30.0 native installation separately. The GenAI core and CUDA add-on remain in the package. The WinML Python wheel now depends on the standard onnxruntime Python package instead of bundling WinML SDK runtime DLLs.