Net4x.HtmlPersist 2.0.0.26283

Net4x.HtmlPersist

Scrapes the <table> of an HTML page into a System.Data.DataTable and persists it to a database: the table is created if missing, rows violating the primary key are dropped, existing rows are deleted by key and re-inserted.

Built on Net4x.BrowserLibrary for page retrieval and Net4x.DapperLibrary for persistence.

Install

dotnet add package Net4x.HtmlPersist

Target frameworks: net461, net5.0-windows, net6.0-windows, net7.0-windows, net8.0-windows. Windows only.

Requirements

A Dapper context must be registered before any persisting call, and a connection string named ConnectionString must be reachable:

using DapperLibrary.SqlServer.DbContexts;

SqlDapperContext.Use();

<connectionStrings>
  <add name="ConnectionString"
       providerName="Microsoft.Data.SqlClient"
       connectionString="Server=localhost;Database=Geography;TrustServerCertificate=True;Trusted_Connection=True;" />
</connectionStrings>

Quick start

using HtmlPersist;

SqlDapperContext.Use();

var rows = DataTablePersister.CreateInstance()
    .PersistTableFromUrl(
        url:              "https://example.com/comuni",
        tableName:        "Comuni",
        emptyTable:       true,
        maximized:        true,
        optionsToAdd:     [],
        keywordsToExclude: ["non ha prodotto risultati"],
        primaryKeys:      "Nr");

Returns the number of rows saved, 0 when the page yielded nothing or a handler cancelled, -1 when persisting threw (the exception is logged).

Get the DataTable without persisting it — extension methods on IBrowserUtility:

using var utility = BrowserUtility.CreateInstance();
var table = await utility.GetDataTableFromUrlAsync(url, maximized: true, optionsToAdd: []);
var other = await utility.GetDataTableFromPageSourceAsync(html, optionsToAdd: []);

API

DataTablePersister : IDataTablePersister

Rewriting a column before it is saved:

var persister = DataTablePersister.CreateInstance();
persister.TableObtained += (_, e) =>
{
    foreach (DataRow row in e.Value.Table.Rows)
        row["Provincia"] = ((string)row["Provincia"]).Replace("-", " ");
};

IDataTableGetter

XmlCleaner : IXmlCleaner

XmlCleaner.Instance.Clean(heading) — normalises a header cell: CRLF/LF and   to spaces, runs of spaces collapsed, < spaced out.

Behaviour notes

• Page source is normalised with ReplaceDiacriticsWithNormalLetters() before parsing.
• When the first parse finds no columns, the page is fetched once more non-maximized before giving up.
• Rows that would violate the primary key are removed, not merged.
• emptyTable: true issues a delete from <table> before inserting.
• PersistTableFromUrl uses a primary-key column size of 500.

Related packages