Themis.Inference.Cuda 4.0.2-preview

Themis.Inference.Cuda

Themis.Inference.Cuda is an optional ONNX Runtime CUDA integration for Themis on Linux x64 and is published with a -preview NuGet version. Register it alongside AddThemis:

services.AddThemis(options => options.Model = "laya-production");
services.AddThemisCuda();

The default configuration requires the CUDAExecutionProvider and fails instead of silently using CPU. To permit CPU fallback when CUDA is unavailable, pass InferenceRuntimeConfiguration with PreferAccelerator and leave RequiredExecutionProviderName unset. When several accelerator providers are registered, set RequiredExecutionProviderName to the desired provider.

Publish the consumer with linux-x64. The package uses the ONNX Runtime 1.31.0 Linux GPU native runtime in place of the CPU native runtime for that RID, while retaining ONNX Runtime's CPU execution provider. Other RIDs keep the CPU runtime and do not gain CUDA support.

Install a compatible NVIDIA driver, CUDA, and cuDNN runtime on the target system. The exact CUDA/cuDNN combination must match the selected Microsoft.ML.OnnxRuntime.Gpu.Linux package. CUDA provider binaries are included by that package; TensorRT provider binaries may also be present, but Themis does not select or configure TensorRT through this integration.

Initial hardware-validation target

The initial target assumption is Ubuntu 22.04 LTS on Linux x64, with an NVIDIA CUDA-capable GPU and a host driver compatible with CUDA 13.0 or newer. ONNX Runtime 1.31.0's Linux GPU package uses CUDA 13.x and cuDNN 9.x; the installed driver, CUDA runtime, and cuDNN libraries must be mutually compatible. The exact GPU model and driver build remain to be recorded when hardware validation is available.

The representative model is the pinned Mattepiu/laya-onnx test bundle at revision a7f385bd51d35f5cbeb88e8bc4f0c7e7c0f24c44. Its CUDA integration request contains three typed questions (choice, score, and noul) in one decision request; sequence length and question batching follow the bundle manifest. This target is a validation assumption, not a hardware result or support claim.

This package has not been validated on Linux GPU hardware. Provider registration or successful inference is not evidence that model nodes ran on CUDA. Verify the intended provider in ONNX Runtime profiles before relying on accelerated execution or making a device/model support claim.

References:

• ONNX Runtime CUDA requirements
• ONNX Runtime Linux installation requirements