BergamotTranslatorSharp.Json 0.7.0

BergamotTranslatorSharp

BergamotTranslatorSharp is a C# wrapper for Bergamot Translator. It allows .NET applications to use an offline machine translation engine.

| EN | JA |

Overview

Bergamot Translator is an offline translation engine. The official website is https://browser.mt/. This library wraps its functionality for use from C#.

Features

• Offline translation capability
• Multi-language support
• Fast processing
• HTML markup preservation
• Terminology dictionary for prescribed translations
• Translation of JSON, YAML, TOML, JSON5, INI, CBOR, and MessagePack string values
• Easy integration with .NET applications

Installation

Install from NuGet

Install-Package BergamotTranslatorSharp

Or:

dotnet add package BergamotTranslatorSharp

Requirements

• .NET 8.0 or later (the library targets .NET 8.0 and .NET 10.0)
• Windows x86, Windows x64, Windows ARM64, Linux x64, macOS x64, or macOS ARM64

Build the native library from source

Windows x86 builds use vcpkg, the Visual Studio Win32 toolchain, and OpenBLAS:

cmake -S . -B out\build\windows-x86-release -A Win32 -DBUILD_ARCH=core2 -DUSE_STATIC_LIBS=ON -DUSE_MKL=OFF -DGIT_SUBMODULE=OFF -DVCPKG_TARGET_TRIPLET=x86-windows-static -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
cmake --build out\build\windows-x86-release --config Release --target bergamot_translator_dynamic
cmake --install out\build\windows-x86-release --prefix libs --component bergamot_translator_dynamic

Windows ARM64 builds use vcpkg and Visual Studio's ARM64 clang-cl toolchain. From an ARM64 developer prompt, set VCPKG_ROOT and run:

set VCPKG_ROOT=C:\vcpkg
cmake --preset windows-arm64-clangcl-release
cmake --build --preset windows-arm64-clangcl-release
cmake --install out\build\windows-arm64-clangcl-release --prefix libs --component bergamot_translator_dynamic

Usage

1. Download models

The old mozilla/firefox-translations-models repository is no longer maintained. Models are now published through Mozilla's official model registry, with the model files hosted under the baseUrl in that registry.

The following example requires curl, jq, and gzip. First, download the registry and inspect the available language directions:

REGISTRY_URL=https://storage.googleapis.com/moz-fx-translations-data--303e-prod-translations-data/db/models.json
curl --fail --location --output models.json "$REGISTRY_URL"
jq -r '.models | keys[]' models.json

Choose a direction and inspect its model candidates. Registry directions use a hyphen, such as de-en for German to English and en-ja for English to Japanese.

DIRECTION=en-ja
jq --arg direction "$DIRECTION" \
  '.models[$direction] | to_entries | map({
    index: .key,
    architecture: .value.architecture,
    releaseStatus: .value.releaseStatus,
    files: .value.files
  })' models.json

Set MODEL_INDEX to the candidate selected from that output. The example below selects index 1, the current Release candidate for en-ja, and stores it in models/enja. Change DIRECTION, MODEL_INDEX, and MODEL_DIR together when using another language direction or candidate.

DIRECTION=en-ja
MODEL_INDEX=1
MODEL_DIR=models/enja
set -euo pipefail
BASE_URL=$(jq -r '.baseUrl' models.json)

mkdir -p "$MODEL_DIR"
jq -r --arg direction "$DIRECTION" --argjson index "$MODEL_INDEX" '
  .models[$direction][$index].files
  | [
      .model.path,
      .vocab.path,
      .srcVocab.path,
      .trgVocab.path,
      .lexicalShortlist.path
    ]
  | .[]
  | select(type == "string")
' models.json |
while IFS= read -r path; do
  curl --fail --location \
    --output "$MODEL_DIR/$(basename "$path")" \
    "$BASE_URL/$path"
done

gzip --decompress "$MODEL_DIR"/*.gz

This downloads the model, its single vocabulary or separate source and target vocabularies, and the lexical shortlist. After decompression, create the configuration file in MODEL_DIR and use the exact decompressed file names shown by ls "$MODEL_DIR".

2. Create a configuration file

Create a config.yml or config.txt file in the same directory as the model files.

Example for an English to Japanese model:

relative-paths: true
models:
- model.enja.intgemm.alphas.bin
vocabs:
- srcvocab.enja.spm
- trgvocab.enja.spm
shortlist:
- lex.50.50.enja.s2t.bin
- false
beam-size: 1
normalize: 1.0
word-penalty: 0
max-length-break: 128
mini-batch-words: 1024
workspace: 128
max-length-factor: 2.0
skip-cost: true
cpu-threads: 0
quiet: true
quiet-translation: true
gemm-precision: int8shiftAlphaAll

Notes:

• The file names in models, vocabs, and shortlist must match the files in the selected model directory.
• If the model file name contains alphas, use gemm-precision: int8shiftAlphaAll.
• Otherwise, use gemm-precision: int8shiftAll.
• When relative-paths: true is used, keep the configuration file and the model files together, or update the paths accordingly.

3. Translate text from C#

The current API uses BlockingService. Pass one or two configuration file paths to the constructor.

using BergamotTranslatorSharp;

var configPath = Path.Combine(
    AppDomain.CurrentDomain.BaseDirectory,
    "models",
    "enja",
    "config.txt");

using var service = new BlockingService(configPath);

var translated = service.Translate("Hello, world!");

Console.WriteLine(translated);

var translatedBatch = service.Translate(["Hello, world!", "How are you?"]);
foreach (var translatedText in translatedBatch)
{
    Console.WriteLine(translatedText);
}

Translate(IEnumerable<string>) translates each text as a separate input and returns results in the same order.

To translate text content while preserving HTML markup, pass true as the second argument:

var translatedHtml = service.Translate("<p>Hello, <strong>world</strong>!</p>", html: true);

var translatedHtmlBatch = service.Translate(
    ["<p>Hello, <strong>world</strong>!</p>", "<p>How are you?</p>"],
    html: true);

Pass a dictionary to prescribe translations for terms in plain text. This works for single and batch translation, including a two-model pivot chain:

var dictionary = new Dictionary<string, string>
{
    ["Mana Reactor"] = "マナリアクター",
    ["Shinra"] = "神羅",
};

var translatedWithDictionary = service.Translate(
    "The Mana Reactor was built by Shinra.", dictionary);
var translatedBatchWithDictionary = service.Translate(
    ["The Mana Reactor was built by Shinra.", "Shinra owns it."], dictionary);

Dictionary keys match exact, case-sensitive substrings. Every occurrence is replaced, and longer keys take priority when matches overlap. Empty keys are rejected. Dictionary translation accepts plain-text input; combining it with caller-provided HTML is not supported.

To translate JSON, YAML, or TOML values, add the package for each format you use. Each package depends only on the core library; the YAML and TOML packages do not depend on the JSON package.

dotnet add package BergamotTranslatorSharp.Json
dotnet add package BergamotTranslatorSharp.Yaml
dotnet add package BergamotTranslatorSharp.Toml

using BergamotTranslatorSharp.Json;
using BergamotTranslatorSharp.Yaml;
using BergamotTranslatorSharp.Toml;

var json = service.TranslateJson("""{"title":"Hello, world!","count":1}""", dictionary);
var yaml = service.TranslateYaml("title: Hello, world!\ncount: 1\n", dictionary);
var toml = service.TranslateToml("title = 'Hello, world!'\ncount = 1\n", dictionary);

The dictionary is optional. Keys, numbers, booleans, nulls, empty strings, and whitespace-only strings are not translated. Each selected string value is translated separately as plain text. For another data format, implement the core ITranslatableDocument<T> interface and call service.Translate(document, dictionary).

For JSON5, INI, CBOR, or MessagePack, install the matching .NET 10 package. Each format package has a dependency on its corresponding REDox parser package; BergamotTranslatorSharp.REDox provides the shared DOM translation API. REDox packages are Apache-2.0 licensed (REDox).

dotnet add package BergamotTranslatorSharp.REDox.Json5
dotnet add package BergamotTranslatorSharp.REDox.Ini
dotnet add package BergamotTranslatorSharp.REDox.Cbor
dotnet add package BergamotTranslatorSharp.REDox.MessagePack
dotnet add package BergamotTranslatorSharp.REDox

using BergamotTranslatorSharp.REDox.Json5;
using BergamotTranslatorSharp.REDox.Ini;
using BergamotTranslatorSharp.REDox.Cbor;
using BergamotTranslatorSharp.REDox.MessagePack;
using BergamotTranslatorSharp.REDox;
using REDox.Json;

var json5 = service.TranslateJson5("{ title: 'Hello, world!' }", dictionary);
var ini = service.TranslateIni("title=Hello, world!", dictionary);
var translatedCbor = service.TranslateCbor(cborBytes, dictionary);
var translatedMessagePack = service.TranslateMessagePack(messagePackBytes, dictionary);

using var document = Json5Document.Parse("{ title: 'Hello, world!' }");
var translatedElement = service.TranslateRedox(document.RootElement, dictionary);

JSON5 comments and trivia are preserved; INI trivia is preserved where REDox exposes it. The REDox document handlers translate nonblank string values, not map keys or non-string values. TranslateRedox accepts an existing DElement and returns a translated clone, leaving the supplied DOM unchanged.

If you pass one configuration file path, BlockingService uses that model directly. If you pass two configuration file paths, the native service uses them as a pivot translation chain.

4. Run the .NET tool

BergamotTranslatorSharp.Tool is a .NET 10 tool with the command name bergamot. After the tool package is published, a .NET 10 SDK can download and run it with dnx:

dnx BergamotTranslatorSharp.Tool -- en ja "Hello, world!"

To use the bergamot command directly in the examples below, install the tool globally:

dotnet tool install --global BergamotTranslatorSharp.Tool

To run the project from this repository:

dotnet run --project BergamotTranslatorSharp.Tool -- en ja "Hello, world!"

The arguments are source language, target language, and text. The tool prefers a Release model from Mozilla's registry, downloads its files, and reuses them from the user application-data directory (BergamotTranslatorSharp/Tool). If a direct model is unavailable, it tries translating through English. To preserve HTML markup, add --html:

bergamot en ja --html "<p>Hello, <strong>world</strong>!</p>"

Translate a supported document with --file. The format is selected from the file extension, or can be specified explicitly with --format. Supported formats are JSON, JSON5, YAML, TOML, INI, HTML, CBOR, and MessagePack. HTML file translation cannot be combined with --dictionary. Use --output to write the result to a file; otherwise it is written to standard output.

bergamot en ja --file input.json
bergamot en ja --file data.bin --format messagepack --output translated.bin

To use a terminology dictionary, provide a UTF-8 CSV with source terms in the first column and prescribed translations in the second. The source,target header is optional. Quote fields that contain commas, newlines, or double quotes; write a quote inside a quoted field as "".

source,target
Mana Reactor,マナリアクター
Shinra,神羅

bergamot en ja --dictionary terms.csv "The Mana Reactor was built by Shinra."

Dictionary input is plain text only and cannot be combined with --html. Duplicate source terms, empty source terms, and records with a column count other than two are rejected.

Run bergamot --help for usage. Model files are downloaded when absent or invalid. The registry metadata is cached for one day and reused if the registry cannot be reached.

Troubleshooting

Failed to create translator instance

This usually means the model could not be loaded. Check the following:

• The configuration file path is correct.
• The model, vocabulary, and shortlist files exist.
• The file names in the configuration file match the actual files.
• gemm-precision matches the model type.
• The native bergamot library can be loaded on your platform.

No translation or unexpected output

Check that the model direction matches the input language. For example, a deen model is intended for German to English input, while an enja model is intended for English to Japanese input.

License

This project is released under the MPL-2.0 license.

Contribution

Please report bugs and feature requests to the GitHub Issue Tracker. Pull requests are also welcome.

Acknowledgments

This project is based on browsermt/bergamot-translator.