WebReaper 10.0.0

WebReaper

Overview

WebReaper is a declarative, high-performance web scraper, crawler and parser in C#. Crawl any web site, parse the data, and save the structured result to a file, a database, or pretty much anywhere you want — with a simple, extensible fluent API.

As of 10.0.0 the core WebReaper package is dependency-light, Native-AOT-ready and Newtonsoft-free: a plain HTTP → file crawl pulls only AngleSharp and Microsoft.Extensions.* (Polly left core in ADR-0026 in favour of the IRetryPolicy seam — the default FixedAttemptsRetryPolicy is hand-rolled). Heavier capabilities (headless browser, MongoDB, Redis, Azure Cosmos DB, Azure Service Bus, SQLite-backed local durable scheduler/tracker, LLM extraction, MCP server, [ScrapeSchema] source generator) ship as optional satellite packages you add only when you need them — see Packages.

Quick start

dotnet add package WebReaper

using WebReaper.Builders;

var engine = await ScraperEngineBuilder
    .Crawl("https://www.alexpavlov.dev/blog")
    .Extract(new()
    {
        new("title", ".text-3xl.font-bold"),
        new("text", ".max-w-max.prose.prose-dark")
    })
    .Follow("a.text-gray-900.transition")
    .WriteToJsonFile("output.json")
    .PageCrawlLimit(10)
    .WithParallelismDegree(30)
    .LogToConsole()
    .BuildAsync();

await engine.RunAsync();

That example is pure HTTP — no browser, no extra packages. For JavaScript-rendered pages, add WebReaper.Puppeteer (see Parsing dynamic pages).

AI-native — the smallest possible call

Since 10.0.0 (the AI-native wave + deepening, ADR-0040..0064), the funnel's no-schema wedge — one page, LLM-ready Markdown, no boilerplate:

var engine = await ScraperEngineBuilder
    .Crawl("https://example.com")
    .AsMarkdown()
    .WriteToConsole()
    .BuildAsync();

await engine.RunAsync();

The CLI mirrors it — the Native-AOT single-binary webreaper (ADR-0043) ships as a prebuilt asset on every tagged GitHub release across six RIDs (linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64, win-arm64). PackAsTool=true is incompatible with PublishAot=true on one target, so a non-AOT dotnet tool install target is deferred to a 10.0.x — file an issue if you need it.

# Latest release, RID = same name dotnet uses (osx-arm64, linux-x64, win-x64, …)
RID=osx-arm64
gh release download --repo pavlovtech/WebReaper --pattern "webreaper-*-$RID.*"
# Linux is .tar.gz; macOS + Windows are .zip
case "$RID" in linux-*) tar -xzf webreaper-*-"$RID".tar.gz ;;
               *)       unzip   webreaper-*-"$RID".zip    ;; esac
cd webreaper-*-"$RID"
./webreaper scrape https://example.com
./webreaper map https://example.com --search /blog/
./webreaper init    # installs the Agent Skill to .claude/skills/webreaper/

The release pages serve direct URLs too — no gh required: https://github.com/pavlovtech/WebReaper/releases/download/<TAG>/webreaper-<TAG>-<RID>.<ext>.

Or build from source — useful between tags or on an unlisted RID:

git clone https://github.com/pavlovtech/WebReaper.git && cd WebReaper
dotnet publish WebReaper.Cli -c Release -r <rid> --self-contained true
./WebReaper.Cli/bin/Release/net10.0/<rid>/publish/WebReaper.Cli scrape https://example.com

Drop in an LLM extractor when the deterministic path can't reach a field (WebReaper.AI satellite — Microsoft.Extensions.AI binding):

using Microsoft.Extensions.AI;
using WebReaper.AI;

var chatClient = /* your IChatClient — OpenAI, Anthropic, Ollama, … */;

var engine = await ScraperEngineBuilder
    .Crawl("https://example.com")
    .Extract(schema)
    .WithLlmFallback(chatClient)        // ADR-0046: det-first → LLM-fallback router
    // or .WithLlmSelfHealing(chatClient)  // ADR-0047: LLM proposes selectors, fold validates, cache demotes back to deterministic
    .WriteToJsonFile("out.jsonl")
    .BuildAsync();

await engine.RunAsync();

Generate the Schema from a typed POCO — [ScrapeSchema], Pydantic-parity that Python's reflection structurally cannot match (WebReaper.Extraction.Generators):

using WebReaper.Extraction.Attributes;

[ScrapeSchema]
public partial class Article
{
    [ScrapeField("h1")]            public string? Title { get; set; }
    [ScrapeField(".views", Type = SchemaFieldType.Integer)]  public int Views { get; set; }
    [ScrapeField(".tag", IsList = true)]  public List<string> Tags { get; set; } = new();
}

// Generated at compile time, reflection-free, AOT-clean:
//   public static Schema Schema { get; }
//   public static Article Materialize(JsonObject json)

var engine = await ScraperEngineBuilder
    .Crawl("https://example.com/post")
    .Extract(Article.Schema)
    .Subscribe(p => HandleArticle(Article.Materialize(p.Data)))
    .BuildAsync();

Runnable end-to-end demo of every AI-native API in this section — Examples/WebReaper.AiNativeShowcase:

dotnet run --project Examples/WebReaper.AiNativeShowcase -- markdown
dotnet run --project Examples/WebReaper.AiNativeShowcase -- map
dotnet run --project Examples/WebReaper.AiNativeShowcase -- sourcegen
dotnet run --project Examples/WebReaper.AiNativeShowcase -- llm
dotnet run --project Examples/WebReaper.AiNativeShowcase -- router
dotnet run --project Examples/WebReaper.AiNativeShowcase -- changetrack

Table of contents

• Install
• Packages
• Requirements
• Features
• Usage examples
• API overview
  • Parsing dynamic pages (SPA)
  • Running JavaScript / page actions
  • Persist the progress locally
  • Authorization
  • How to disable headless mode
  • Cleaning data from a previous run
  • Distributed and serverless scraping
  • Storage and scheduler backends
  • Extensibility: adding a sink
  • Interfaces
  • Main entities
• Repository structure
• License

Install

Core package — HTTP crawling/parsing, in-memory and file-backed state, Console/CSV/JSON-Lines sinks:

dotnet add package WebReaper

Add a satellite only for the capability you need (each brings its own SDK so the core stays light):

dotnet add package WebReaper.Puppeteer        # headless-browser (SPA / JS) pages
dotnet add package WebReaper.Mongo            # MongoDB sink + config/cookie storage
dotnet add package WebReaper.Redis            # Redis scheduler, tracker, sink, storage
dotnet add package WebReaper.AzureServiceBus  # Azure Service Bus distributed scheduler
dotnet add package WebReaper.Cosmos           # Azure Cosmos DB sink
dotnet add package WebReaper.Sqlite           # SQLite local durable scheduler + visited-link tracker

Packages

All eleven NuGet packages are versioned in lockstep at 10.0.0 (1 core + 10 satellites). Core and the satellites move together in release waves (ADR-0022 → 8.0.0, ADR-0023 → 9.0.0, ADR-0025 + ADR-0040..0064 → 10.0.0); WebReaper.Sqlite, added at 7.1.0, joined the lockstep from 8.0.0; WebReaper.AI, WebReaper.Mcp, WebReaper.Extraction.Attributes, and WebReaper.Extraction.Generators joined at 10.0.0 (the AI-native wave). All packages are MIT-licensed (ADR-0017; relicensed from GPL-3.0-or-later in the 10.0.0 wave), and every satellite wires itself in through the builder's public registration seam. WebReaper.Cli (ADR-0043, the AOT single-binary agent surface) is not a NuGet package in 10.0.0 — build from source per Quick start.

The core default page loader is HTTP-only. Crawling a dynamic page (CrawlWithBrowser / FollowWithBrowser / PaginateWithBrowser) without WebReaper.Puppeteer registered throws an InvalidOperationException telling you to add the package and call .WithPuppeteerPageLoader().

Requirements

.NET 10. The core package is IsAotCompatible — it Native-AOT-publishes with zero trim/AOT warnings (proven by the AOT smoke test in CI). Satellites carry their own SDK dependencies and are not AOT-clean by design; reference one only when you use it.

Features

• ⚡ High crawling speed through parallelism and asynchrony
• 🗒 Declarative and easy to use
• 🪶 Dependency-light, Native-AOT-ready, Newtonsoft-free core
• 💾 Console, CSV and JSON-Lines sinks out of the box; MongoDB, Redis and Azure Cosmos DB via satellites
• 🌎 Scalable: run on cloud VMs, serverless functions or on-prem; go distributed with Redis or Azure Service Bus
• 🐙 Crawl and parse Single Page Applications with Puppeteer (WebReaper.Puppeteer)
• 🖥 Proxy support
• 🌀 Extensible: replace any out-of-the-box seam with your own implementation

Usage examples

• Data mining
• Gathering data for machine learning
• Online price-change monitoring and price comparison
• News aggregation
• Product-review scraping (to watch the competition)
• Tracking online presence and reputation

API overview

Parsing dynamic pages (SPA)

Parsing Single Page Applications is simple: use CrawlWithBrowser and/or FollowWithBrowser, add the WebReaper.Puppeteer package, and register it with .WithPuppeteerPageLoader(). Puppeteer then loads those pages in a headless browser.

dotnet add package WebReaper.Puppeteer

using WebReaper.Builders;
using WebReaper.Puppeteer;

var engine = await ScraperEngineBuilder
    .CrawlWithBrowser("https://www.alexpavlov.dev/blog")
    .Extract(new()
    {
        new("title", ".text-3xl.font-bold"),
        new("text", ".max-w-max.prose.prose-dark")
    })
    .WithPuppeteerPageLoader()
    .FollowWithBrowser("a.text-gray-900.transition")
    .WriteToJsonFile("output.json")
    .PageCrawlLimit(10)
    .WithParallelismDegree(30)
    .LogToConsole()
    .BuildAsync();

await engine.RunAsync();

.WithPuppeteerPageLoader() is parameterless and reproduces the pre-7.0 behaviour exactly (one shared cookie container, optional proxy applied the browser's own way). The first dynamic-page run downloads Chromium via Puppeteer.

Running JavaScript / page actions

You can run JavaScript and drive the page as it loads in the headless browser. Pass an actions lambda (e.g. .ScrollToEnd()) — useful when the content you need appears only after clicks, scrolls, etc.

using WebReaper.Builders;
using WebReaper.Puppeteer;

var engine = await ScraperEngineBuilder
    .CrawlWithBrowser("https://www.reddit.com/r/dotnet/", actions => actions
        .ScrollToEnd()
        .Build())
    .Extract(new()
    {
        new("title", "._eYtD2XCVieq6emjKBH3m"),
        new("text", "._3xX726aBn29LDbsDtzr_6E._1Ap4F5maDtT1E1YuCiaO0r.D3IL3FD0RFy_mkKLPwL4")
    })
    .Follow("a.SQnoC3ObvgnGjWt90zD9Z._2INHSNB8V5eaWp4P0rY_mE")
    .WithPuppeteerPageLoader()
    .WriteToJsonFile("output.json")
    .LogToConsole()
    .BuildAsync();

await engine.RunAsync();

Console.ReadLine();

PageActionBuilder exposes Click, Wait, ScrollToEnd, WaitForSelector, WaitForNetworkIdle, EvaluateExpression, Repeat/RepeatWithDelay, and Build().

Persist the progress locally

To persist the job queue and visited links locally — so you can resume where you left off — use WithTextFileScheduler and TrackVisitedLinksInFile:

using WebReaper.Builders;

var engine = await ScraperEngineBuilder
    .Crawl("https://rutracker.org/forum/index.php?c=33")
    .Extract(new()
    {
        new("name", "#topic-title"),
        new("category", "td.nav.t-breadcrumb-top.w100.pad_2>a:nth-child(3)"),
        new("subcategory", "td.nav.t-breadcrumb-top.w100.pad_2>a:nth-child(5)"),
        new("torrentSize", "div.attach_link.guest>ul>li:nth-child(2)"),
        new("torrentLink", ".magnet-link", "href"),
        new("coverImageUrl", ".postImg", "src")
    })
    .WithLogger(logger)
    .Follow("#cf-33 .forumlink>a")
    .Follow(".forumlink>a")
    .Paginate("a.torTopic", ".pg")
    .WriteToJsonFile("result.json")
    .IgnoreUrls(blackList)
    .WithTextFileScheduler("jobs.txt", "currentJob.txt")
    .TrackVisitedLinksInFile("links.txt")
    .BuildAsync();

The file scheduler is the zero-dependency default: an append-only job file, a 300 ms poll and a sidecar position file. For a long single-machine crawl that must survive kill -9 and resume by query — without standing up a Redis server — add the WebReaper.Sqlite satellite and swap the two local backends. "Resume" becomes a SELECT over an indexed table; there is no position file to keep in sync (the visited-link table is the set — no in-memory mirror). The core file adapters are unchanged and stay the default; this is opt-in:

using WebReaper.Builders;
using WebReaper.Sqlite;   // dotnet add package WebReaper.Sqlite

var engine = await ScraperEngineBuilder
    .Crawl("https://rutracker.org/forum/index.php?c=33")
    .Extract(new() { new("name", "#topic-title") })
    .Follow(".forumlink>a")
    .Paginate("a.torTopic", ".pg")
    .WriteToJsonFile("result.json")
    .WithSqliteScheduler("crawl/state.db")        // resume is a query, not a position file
    .TrackVisitedLinksInSqlite("crawl/state.db")  // the table is the set
    .BuildAsync();

Pass dataCleanupOnStart: true to either call to start a fresh crawl (clears that table at start).

Authorization

If the site needs authorization, call SetCookies and fill the CookieContainer with the cookies required. You perform the login yourself; WebReaper only uses the cookies you provide.

using System.Net;
using WebReaper.Builders;

var engine = await ScraperEngineBuilder
    .Crawl("https://rutracker.org/forum/index.php?c=33")
    .Extract(new() { new("name", "#topic-title") })
    .WithLogger(logger)
    .SetCookies(cookies =>
    {
        cookies.Add(new Cookie("AuthToken", "123"));
    })
    // ...
    .BuildAsync();

How to disable headless mode

When scraping with a browser (CrawlWithBrowser / FollowWithBrowser, via WebReaper.Puppeteer) the default is headless — you don't see the browser. Seeing it can help with debugging; disable headless mode with .HeadlessMode(false):

using WebReaper.Builders;
using WebReaper.Puppeteer;

var engine = await ScraperEngineBuilder
    .CrawlWithBrowser("https://www.reddit.com/r/dotnet/", actions => actions
        .ScrollToEnd()
        .Build())
    .Extract(new() { new("title", "._eYtD2XCVieq6emjKBH3m") })
    .HeadlessMode(false)
    .WithPuppeteerPageLoader()
    // ...
    .BuildAsync();

Cleaning data from a previous run

To start fresh, pass dataCleanupOnStart: true to the relevant builder method.

// Result file — note: WriteToJsonFile already defaults dataCleanupOnStart to TRUE
.WriteToJsonFile("output.json", dataCleanupOnStart: true)

// Visited-link tracker
.TrackVisitedLinksInFile("visited.txt", dataCleanupOnStart: true)

// Job queue / scheduler
.WithTextFileScheduler("jobs.txt", "currentJob.txt", dataCleanupOnStart: true)

The dataCleanupOnStart parameter exists on the satellite sinks too (e.g. WriteToMongoDb, WriteToRedis, WriteToCosmosDb). Note WriteToJsonFile defaults it to true (it wipes the file on start) — the opposite of the other sinks, which default to false. The "JSON" file sink writes JSON Lines (one compact JSON object per line), not a JSON array.

Distributed and serverless scraping

Swap the scheduler, config storage and link tracker to Redis or Azure Service Bus and multiple workers / serverless functions can share one crawl. Examples/WebReaper.AzureFuncs shows the serverless shape with two functions:

• StartScraping builds the scraper configuration, seeds the distributed Outstanding-work latch, and enqueues the first job (the start URL) onto the queue (e.g. Azure Service Bus).
• WebReaperSpider is the distributed Crawl driver, triggered by each queued job. It gets a bare ISpider from new DistributedSpiderBuilder()...BuildSpider() (load → Crawl step → JobReport), then interprets the report: an atomic visited-link test-and-set gates duplicates/redeliveries, a parsed page is fanned out to the sink, discovered child jobs are enqueued back onto the queue, and a distributed Outstanding-work latch detects when all work has drained. It never throws to signal the crawl limit, so the queue is never poisoned (ADR-0022).

DistributedSpiderBuilder.BuildSpider() (the ADR-0009 distributed-worker seam) returns an ISpider without building or persisting a ScraperConfig; it has no Crawl seed and no BuildAsync — the worker's config is persisted separately by the start endpoint (ScraperEngineBuilder.Crawl(...).Extract(...).Build()) and read from storage at crawl time. This is the "two seams, not one bug" split (ADR-0025). See also Examples/WebReaper.DistributedScraperWorkerService.

Storage and scheduler backends

Every backend is a swappable seam. In-memory is the default; file-backed lives in core; the rest come from satellites.

Extensibility: adding a sink

Out of the box the core package sends parsed data to the Console, CSV and JSON-Lines sinks; MongoDB, Redis and Cosmos DB sinks come from satellites. Add your own by implementing IScraperSink:

using WebReaper.Sinks.Abstract;
using WebReaper.Sinks.Models;

public interface IScraperSink
{
    bool DataCleanupOnStart { get; set; }
    Task EmitAsync(ParsedData entity, CancellationToken cancellationToken = default);
}

ParsedData is record ParsedData(string Url, JsonObject Data) — Data is a System.Text.Json.Nodes.JsonObject (no Newtonsoft). A minimal console sink:

using System.Text.Json.Nodes;
using WebReaper.Sinks.Abstract;
using WebReaper.Sinks.Models;

public class ConsoleSink : IScraperSink
{
    public bool DataCleanupOnStart { get; set; }

    public Task EmitAsync(ParsedData entity, CancellationToken cancellationToken = default)
    {
        Console.WriteLine(entity.Data.ToJsonString());
        return Task.CompletedTask;
    }
}

Register it with AddSink:

using WebReaper.Builders;

var engine = await ScraperEngineBuilder
    .Crawl("https://rutracker.org/forum/index.php?c=33")
    .Extract(new()
    {
        new("name", "#topic-title"),
    })
    .AddSink(new ConsoleSink())
    .Follow("#cf-33 .forumlink>a")
    .Follow(".forumlink>a")
    .Paginate("a.torTopic", ".pg")
    .BuildAsync();

For result callbacks without a custom sink, use .Subscribe(Action<ParsedData>) or .PostProcess(Func<Metadata, JsonObject, Task>).

Interfaces

Main entities

• Job — a record representing one unit of work for the spider.
• LinkPathSelector — a selector for links to be crawled.
• CrawlOutcome — the closed result of a crawl step: a parsed target page, followed links, or paginated pages.
• Schema fold — the single recursive Schema interpreter (SchemaFold<TNode>); every backend reuses it instead of re-implementing the walk.

Repository structure

License

WebReaper is licensed under the MIT License (ADR-0017). All eleven NuGet packages (core + ten satellites) plus the WebReaper.Cli project ship under the same terms. Use it commercially, embed it in proprietary software, fork it, modify it, redistribute it; the only ask is that you keep the copyright notice.

Prior to the 10.0.0 wave, WebReaper was GPL-3.0-or-later. The relicense is strictly more permissive: every existing user is unaffected; new users who couldn't embed under GPL now can. Historical contributors are credited in CONTRIBUTORS.md. See docs/adr/0017-relicense-gpl-mit.md for the analysis and contributor consent path.

Contributions are welcome under the same MIT terms; sign-off via DCO (CONTRIBUTING.md).