SystemHarness.Windows 0.29.1

system-harness

A computer-use primitives library for .NET — eyes, hands, and shell in one harness.

system-harness provides a unified interface for programmatic and interactive computer control. It wraps shell execution, process management, filesystem operations, screen capture, OCR, input simulation, UI automation, and document processing into a single coherent API.

No AI inside. No opinions about your agent framework. Just the primitives you need to use a computer — whether you're a bot or a human writing automation scripts.

Why

AI agents need to operate computers. The "brain" (LLM) decides what to do. But it still needs:

• Shell — run cmd, powershell, bash commands and get results
• Process — launch, kill, and list running programs
• FileSystem — read, write, move, delete files and directories
• Screen — capture what's on screen (for vision-capable models or logging)
• Mouse / Keyboard — click, type, drag when there's no API and GUI is the only way
• OCR — read text from screen regions without requiring vision models
• UI Automation — interact with UI elements by accessibility tree, not pixel coordinates
• Office Documents — read and write Word, Excel, PowerPoint, HWP without Office installed

Today, you stitch together 5+ libraries to get all of this. system-harness is one library, one interface, three layers:

Layer 1: Programmatic Control (fast, precise, preferred)
  Shell, Process, FileSystem, Window, Clipboard, Display, SystemInfo

Layer 2: Vision + Action (when GUI is the only way)
  Screen, Mouse, Keyboard, OCR, UIAutomation, TemplateMatcher, DialogHandler

Layer 3: App Automation (document processing)
  Office (Word, Excel, PowerPoint, HWP) — no installation required

Use Layer 1 whenever possible. Fall back to Layer 2 when you must.

Quick Start

dotnet add package SystemHarness.Core
dotnet add package SystemHarness.Windows  # Windows implementation

using SystemHarness;
using SystemHarness.Windows;

using var harness = new WindowsHarness();

// Layer 1 — Programmatic
var result = await harness.Shell.RunAsync("cmd", "/C echo Hello!");
Console.WriteLine(result.StdOut);   // "Hello!\r\n"

await harness.FileSystem.WriteAsync("hello.txt", "world");
var content = await harness.FileSystem.ReadAsync("hello.txt");

await harness.Process.StartAsync("notepad.exe");
await harness.Window.FocusAsync("Notepad");

// Layer 2 — Vision + Action
var screenshot = await harness.Screen.CaptureAsync();
// screenshot.Base64, screenshot.Width, screenshot.Height, screenshot.MimeType

await harness.Mouse.ClickAsync(350, 200);
await harness.Keyboard.TypeAsync("Hello World");
await harness.Keyboard.HotkeyAsync(default, Key.Ctrl, Key.S);

// OCR — read text from screen
var ocrResult = await harness.Ocr.RecognizeScreenAsync();
Console.WriteLine(ocrResult.Text);

// UI Automation — interact with elements by name
var tree = await harness.UIAutomation.GetTreeAsync("Notepad");
await harness.UIAutomation.TypeIntoAsync("Notepad", "Edit", "Hello from automation");

NuGet Packages

MCP Server (AI Tool Integration)

system-harness includes a built-in Model Context Protocol server with 172 commands across 25 categories, accessed through 3 MCP tools using a command dispatch pattern.

Installation

Download the latest release from GitHub Releases and extract it to a directory of your choice.

Configuration

Claude Desktop / Claude Code (claude_desktop_config.json or .mcp.json):

{
  "mcpServers": {
    "system-harness": {
      "command": "C:/path/to/SystemHarness.Mcp.exe"
    }
  }
}

Operator settings — pass as arguments ("args": ["--rate-limit=5"]). The agent can tighten them but not loosen them:

From source (development):

{
  "mcpServers": {
    "system-harness": {
      "command": "dotnet",
      "args": ["run", "--project", "src/SystemHarness.Mcp"]
    }
  }
}

3 MCP Tools

Instead of 172 individual tool definitions (which consume ~12,000 tokens per API call), commands are accessed through 3 dispatch tools:

Command Categories

Compound Facades (reduce multiple tool calls to one)

Office Documents

Read and write Microsoft Office and Korean HWP documents without Office installed — uses OpenXML and OWPML directly.

using SystemHarness.Apps.Office;

// Register in DI
services.AddOfficeReaders();

// Read documents to markdown
var wordContent = await documentReader.ReadWordAsync("report.docx");
var excelContent = await documentReader.ReadExcelAsync("data.xlsx");
var pptxContent = await documentReader.ReadPowerPointAsync("slides.pptx");
var hwpContent = await hwpReader.ReadHwpxAsync("document.hwpx");

// Write documents from structured content
await documentReader.WriteWordAsync("output.docx", documentContent);
await documentReader.WriteExcelAsync("output.xlsx", spreadsheetContent);

// Find and replace
await documentReader.FindReplaceWordAsync("template.docx", new() { { "{{name}}", "John" } });

Safety Features

using var harness = new WindowsHarness(new HarnessOptions
{
    // Block dangerous commands (format, shutdown, rm -rf, etc.)
    CommandPolicy = CommandPolicy.CreateDefault(),

    // Record all actions for auditing
    AuditLog = new InMemoryAuditLog(),
});

// This throws CommandPolicyException:
await harness.Shell.RunAsync("format", "C: /FS:NTFS");

Emergency Stop

var stop = new EmergencyStop();
// Pass stop.Token to any CancellationToken parameter
// Call stop.Trigger() to cancel all operations at once

MCP Server Safety

Every MCP command passes one gate before it runs. A refused command is not run, and the refusal is recorded in the action history:

• Emergency stop — safety.emergency_stop or the operator hotkey (Ctrl+Shift+Escape) cancels running commands and refuses every later action (emergency_stopped). The agent can resume its own stop with safety.resume; a hotkey stop holds until the server restarts.
• Command policy — blocked programs in shell commands and process starts are refused (policy_blocked), including programs chained inside the command part of a shell host (cmd /c, pwsh -Command / -EncodedCommand, bash -c). The arguments a host passes to a script (pwsh -File run.ps1 format, bash run.sh shutdown) are the script's data and are not read as commands; blocked patterns still apply to the whole line.
• Rate limiting — actions over the limit are refused (rate_limited).
• Safe zones — with a zone set, input actions (mouse, keyboard, UI automation, vision clicks, dialogs, window changes) must target the zone window: coordinates inside it (or inside its region), window arguments naming it, keyboard input only while it is in the foreground (outside_safe_zone). A zone window that cannot be found refuses the action; commands whose target is only known after they run (vision.click_text, record.replay) are refused while a zone is set.
• Confirmation — safety.confirm_before writes a JSON request the user approves or denies by editing its status; the agent polls with safety.check_confirmation and has no command to answer its own request.
• Action history — full audit trail of tool invocations, including refusals
• Server files — screenshots, clipboard images and confirmation requests are written to a private per-process directory (%TEMP%\system-harness\<pid>) that is deleted when the server stops; file commands cannot write, move or delete anything inside it (protected_path)

Monitoring

Background monitors track system changes in real-time, writing events to JSONL files:

Dependency Injection

services.AddSystemHarness(); // default options
// or
services.AddSystemHarness(new HarnessOptions
{
    CommandPolicy = CommandPolicy.CreateDefault(),
});

// Inject IHarness or individual services:
public class MyService(IShell shell, IScreen screen, IOcr ocr) { }

Platform Factory

// Auto-detect platform at runtime
using var harness = HarnessFactory.Create();

Windows only. SystemHarness.Windows is the one platform implementation; there is no Linux or macOS implementation and none is planned. On another OS HarnessFactory.Create() throws PlatformNotSupportedException saying so. The platform-independent packages (SystemHarness.Core models, SystemHarness.Apps.*) still work there.

Architecture

SystemHarness.Core              Interfaces + models (zero platform dependencies)
  |
  +-- SystemHarness.Windows     Win32/DXGI/SendInput/FlaUI (Windows implementation)
  |
  +-- SystemHarness.Apps.Office OpenXML/OWPML document processing
  +-- SystemHarness.Apps.Email  IMAP/SMTP via MailKit
  +-- SystemHarness.Apps.Browser Playwright-based web automation
  |
  +-- SystemHarness.Mcp         MCP server (3 tools, 172 commands)

IHarness Services (15 interfaces)

Windows Implementation Details

Design Principles

• Zero AI dependency — no LLM SDKs, no model opinions, no agent loops
• AI-ready — Screenshot returns Base64 for vision APIs; ShellResult is structured
• Layer 1 first — programmatic control is always preferred; vision+action is the fallback
• One interface, multiple backends — same IHarness across all platforms
• Async-first — every operation is Task-based
• No magic — thin wrappers over OS primitives, not a framework
• Safety built-in — command policy, audit logging, emergency stop, safe zones

Roadmap

• Core interfaces and models (15 services)
• Windows Layer 1: Shell, Process, FileSystem, Window, Clipboard, Display, SystemInfo
• Windows Layer 2: Screen (DXGI+GDI), Mouse, Keyboard, OCR, UIAutomation
• Template matching (NCC-based image search)
• Compound facades: vision_click_text, app_open/close, report_get_screen
• Smart waiting: vision_wait_text, ui_wait_element, window_wait, vision_wait_change
• Action verification: vision_click_and_verify, vision_type_and_verify
• Background monitors: file, process, window, clipboard, screen, dialog
• Safety: EmergencyStop, safe zones, rate limiting, confirmation gates
• Session management: save, compare, bookmark
• MCP server with 172 commands (3-tool dispatch architecture)
• Office document processing (Word, Excel, PowerPoint, HWP)
• DPI-aware coordinates, Unicode support, cursor overlay
• NuGet packaging with SourceLink

License

MIT