WebSpark.HttpClientUtility.Crawler 2.1.2

WebSpark.HttpClientUtility.Crawler

Web crawling extension for WebSpark.HttpClientUtility.

Overview

This package provides enterprise-grade web crawling capabilities with robots.txt compliance, HTML parsing, sitemap generation, and real-time progress tracking via SignalR.

Important: This is an extension package. You must also install the base package WebSpark.HttpClientUtility version 2.0.0.

Features

• SiteCrawler: Full-featured web crawler with configurable depth and URL filtering
• SimpleSiteCrawler: Lightweight crawler for basic crawling needs
• Robots.txt Compliance: Automatic parsing and enforcement of robots.txt rules
• HTML Parsing: Extract links, images, and metadata using HtmlAgilityPack
• Sitemap Generation: Generate sitemaps in XML and Markdown formats
• CSV Export: Export crawl results to CSV for analysis
• SignalR Progress: Real-time crawl progress updates via SignalR hub
• Performance Tracking: Built-in metrics for crawl operations

Installation

Install both packages:

dotnet add package WebSpark.HttpClientUtility
dotnet add package WebSpark.HttpClientUtility.Crawler

Quick Start

1. Register Services

using WebSpark.HttpClientUtility;
using WebSpark.HttpClientUtility.Crawler;

var builder = WebApplication.CreateBuilder(args);

// Register base package (required)
builder.Services.AddHttpClientUtility();

// Register crawler package
builder.Services.AddHttpClientCrawler();

var app = builder.Build();

// Optional: Register SignalR hub for progress updates
app.MapHub<CrawlHub>("/crawlHub");

app.Run();

2. Basic Crawling

public class CrawlerService
{
    private readonly ISiteCrawler _crawler;

    public CrawlerService(ISiteCrawler crawler)
    {
        _crawler = crawler;
    }

    public async Task<CrawlResult> CrawlWebsiteAsync(string url)
    {
        var options = new CrawlerOptions
        {
            StartUrl = url,
            MaxDepth = 3,
            MaxPages = 100,
            RespectRobotsTxt = true
        };

        var result = await _crawler.CrawlAsync(options);
        
        Console.WriteLine($"Crawled {result.TotalPages} pages in {result.Duration}");
        return result;
    }
}

3. SimpleSiteCrawler

For lightweight crawling without full recursion:

public class SimpleCrawlerService
{
    private readonly SimpleSiteCrawler _simpleCrawler;

    public SimpleCrawlerService(SimpleSiteCrawler simpleCrawler)
    {
        _simpleCrawler = simpleCrawler;
    }

    public async Task<List<string>> GetAllLinksAsync(string url)
    {
        var result = await _simpleCrawler.CrawlAsync(url);
        return result.DiscoveredUrls;
    }
}

4. Advanced Configuration

builder.Services.AddHttpClientCrawler(options =>
{
    options.DefaultMaxDepth = 5;
    options.DefaultMaxPages = 500;
    options.DefaultTimeout = TimeSpan.FromSeconds(30);
    options.UserAgent = "MyBot/1.0";
});

5. CSV Export

var result = await _crawler.CrawlAsync(options);

await result.ExportToCsvAsync("crawl-results.csv");

6. SignalR Progress Updates

// Client-side JavaScript
const connection = new signalR.HubConnectionBuilder()
    .withUrl("/crawlHub")
    .build();

connection.on("CrawlProgress", (progress) => {
    console.log(`Progress: ${progress.pagesProcessed}/${progress.totalPages}`);
});

await connection.start();

CrawlerOptions

CrawlResult Properties

• TotalPages: Number of pages successfully crawled
• FailedPages: Number of pages that failed to crawl
• Duration: Total time taken for the crawl
• DiscoveredUrls: List of all URLs discovered
• SitemapXml: Generated sitemap in XML format
• SitemapMarkdown: Generated sitemap in Markdown format
• Errors: List of errors encountered during crawl

Robots.txt Support

The crawler automatically:

• Downloads and parses robots.txt from target sites
• Respects Disallow directives
• Honors Crawl-delay settings
• Supports wildcard patterns

Disable with:

options.RespectRobotsTxt = false; // Not recommended

Performance Tips

1. Limit Scope: Use MaxDepth and MaxPages to avoid overwhelming servers
2. Respect Crawl-Delay: The crawler automatically honors robots.txt delays
3. Use AllowedDomains: Restrict crawling to prevent following external links
4. Monitor Progress: Use SignalR hub to track crawl status in real-time

Requirements

• Base Package: WebSpark.HttpClientUtility 2.0.0 (exact match required)
• .NET Version: .NET 8 LTS or .NET 9
• ASP.NET Core: Required for SignalR features

Migration from v1.x

If you're upgrading from WebSpark.HttpClientUtility v1.x:

1. Install the crawler package: dotnet add package WebSpark.HttpClientUtility.Crawler
2. Add using directive: using WebSpark.HttpClientUtility.Crawler;
3. Add DI registration: services.AddHttpClientCrawler();

All crawler APIs remain unchanged - only the registration is different.

Documentation

• Full Documentation
• Migration Guide
• API Reference

License

MIT License - see LICENSE

Support

• GitHub Issues
• NuGet Package