Spidey 6.0.1

dotnet add package Spidey --version 6.0.1                
NuGet\Install-Package Spidey -Version 6.0.1                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Spidey" Version="6.0.1" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Spidey --version 6.0.1                
#r "nuget: Spidey, 6.0.1"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Spidey as a Cake Addin
#addin nuget:?package=Spidey&version=6.0.1

// Install Spidey as a Cake Tool
#tool nuget:?package=Spidey&version=6.0.1                

Spidey

.NET Publish

Library to help with crawling web content. Compatible with .Net Core and .Net Framework.

Setting up the Library

Spidey needs to be added to your app's service collection in order to be wired up properly. This is done by calling the RegisterSpidey extension method on the ServiceCollection object. This is done like so:

new ServiceCollection().RegisterSpidey();

Or if you're using Canister:

new ServiceCollection().AddCanisterModules();

The RegisterSpidey function is an extension method that registers it with the IoC container. When this is done, Spidey is ready to use.

Basic Usage

Spidey really boils down to using one class called Crawler:

ServiceCollection.AddSingleton(new Options
			{
			    ItemFound = FoundFile=>{}                                            //The callback method used when a new page is discovered.
			    Allow = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Regexes of what sites/pages are allowed to be crawled.
			    FollowOnly = new List<string> { "..." },                                 //Regexes of pages to only follow links that are found on them.
			    Ignore = new List<string> { "..." },                                     //Regexes that the system will ignore when they are encountered.
			    StartLocations = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Starting URLs for the crawler.
			    UrlReplacements = new Dictionary<string,string> {...}                    //When the system hits one of the keys in the dictionary, it will replace it with the value.
			});
			

Note that it is recommended that you actually register the Options object in your ServiceCollection and resolve the Crawler object from the service provider but it is not required. You can simply new up an instance of Crawler if you want. Anyway, the Options class has a number of properties, some of which are not displayed above such as NetworkCredentials, UseDefaultCredentials, and Proxy. The callback method is what will be called by the system once a link's info has been received and looks like this:

void CallbackMethod(ResultFile obj) { ... }

The library will handle parsing of links found within the page, downloading the content, etc. for the most part. At this point all you have to do is call the StartCrawl method:

MyCrawler.StartCrawl();

Customization

Note that it's possible to customize the crawler's various parts. The system is divided into the following sections:

  1. Content Parser (IContentParser) - This parses the resulting data and converts it to the ResultFile object.
  2. Engine (IEngine) - This downloads the content from the server.
  3. Link Discoverer (ILinkDiscoverer) - Takes the content from the engine and looks for links to other resources.
  4. Processor (IProcessor) - Takes the parsed content and hands it off to your code. The default one simply calls the method provided in the options.
  5. Scheduler (IScheduler) - Handles handing out work to the various workers.
  6. Pipeline (IPipeline) - Manages the various parts of the process by feeding the content to the next bit of the process.

These subsystems all implement interfaces found in the Spidey.Engines.Interfaces namespace. In order to replace the default in any of these systems all you need to do is create a class that implements the interface that you want to replace. After that the system will automatically pick it up if resolved from the service provider. If you, instead, new up a Crawler object then you will need to compose the Pipeline object.

Installation

The library is available via Nuget with the package name "Spidey". To install it run the following command in the Package Manager Console:

Install-Package Spidey

FAQ

  1. Is it possible to run the crawler using multiple nodes?

The default scheduler assumes that you are only running the crawler from one location and doesn't talk to other instances of the application. But it is possible to replace the scheduler with one that will talk via some mechanism like a database to coordinate work between instances and is recommended for more complex setups.

Build Process

In order to build the library you will require the following:

  1. Visual Studio 2022

Other than that, just clone the project and you should be able to load the solution and build without too much effort.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
6.0.1 84 12/9/2024
6.0.0 106 11/25/2024
5.0.131 94 11/12/2024
5.0.130 83 11/11/2024
5.0.129 83 11/6/2024
5.0.128 85 11/5/2024
5.0.127 78 11/4/2024
5.0.126 83 10/31/2024
5.0.125 72 10/30/2024
5.0.124 83 10/29/2024
5.0.123 92 10/11/2024
5.0.122 86 10/10/2024
5.0.121 80 10/9/2024
5.0.120 92 10/2/2024
5.0.119 92 10/1/2024
5.0.118 89 9/24/2024
5.0.117 103 9/17/2024
5.0.116 142 9/10/2024
5.0.115 99 9/3/2024
5.0.114 100 8/30/2024
5.0.113 109 8/27/2024
5.0.112 107 8/26/2024
5.0.111 126 8/23/2024
5.0.110 117 8/21/2024
5.0.109 113 8/20/2024
5.0.108 118 8/16/2024
5.0.107 115 8/15/2024
5.0.106 97 8/5/2024
5.0.105 81 8/2/2024
5.0.104 91 8/1/2024
5.0.103 92 7/26/2024
5.0.102 107 7/11/2024
5.0.101 106 7/2/2024
5.0.100 111 6/27/2024
5.0.99 91 6/26/2024
5.0.98 121 6/19/2024
5.0.97 109 6/18/2024
5.0.96 118 6/17/2024
5.0.95 118 6/14/2024
5.0.94 101 6/13/2024
5.0.93 100 6/12/2024
5.0.92 113 5/31/2024
5.0.91 105 5/30/2024
5.0.90 107 5/17/2024
5.0.89 108 5/16/2024
5.0.88 130 5/8/2024
5.0.87 130 5/7/2024
5.0.86 131 5/6/2024
5.0.85 88 5/3/2024
5.0.84 96 5/2/2024
5.0.83 107 5/1/2024
5.0.82 130 4/30/2024
5.0.81 112 4/16/2024
5.0.80 113 4/12/2024
5.0.79 110 4/11/2024
5.0.78 132 4/1/2024
5.0.77 121 3/29/2024
5.0.76 123 3/18/2024
5.0.75 118 3/15/2024
5.0.74 120 3/14/2024
5.0.73 121 3/11/2024
5.0.72 114 3/8/2024
5.0.71 116 3/7/2024
5.0.70 136 3/6/2024
5.0.69 125 3/5/2024
5.0.68 126 3/4/2024
5.0.67 125 2/29/2024
5.0.66 113 2/28/2024
5.0.65 123 2/26/2024
5.0.64 122 2/23/2024
5.0.63 127 2/22/2024
5.0.62 129 2/21/2024
5.0.61 119 2/16/2024
5.0.60 118 2/15/2024
5.0.59 129 2/12/2024
5.0.58 120 2/8/2024
5.0.57 115 2/7/2024
5.0.56 116 2/6/2024
5.0.55 106 2/1/2024
5.0.54 121 1/31/2024
5.0.53 117 1/30/2024
5.0.52 111 1/24/2024
5.0.51 114 1/23/2024
5.0.50 125 1/12/2024
5.0.49 125 1/11/2024
5.0.48 127 12/26/2023
5.0.47 120 12/22/2023
5.0.46 106 12/18/2023
5.0.45 101 12/15/2023
5.0.44 97 12/14/2023
5.0.43 113 12/13/2023
5.0.42 108 12/12/2023
5.0.41 154 11/24/2023
5.0.40 132 11/21/2023
5.0.39 118 11/20/2023
5.0.38 124 11/17/2023
5.0.37 116 11/16/2023
5.0.36 123 11/14/2023
5.0.35 124 11/8/2023
5.0.34 108 11/7/2023
5.0.33 127 11/6/2023
5.0.32 123 11/1/2023
5.0.31 117 10/31/2023
5.0.30 133 10/30/2023
5.0.29 128 10/26/2023
5.0.28 153 10/12/2023
5.0.27 154 10/5/2023
5.0.26 141 9/26/2023
5.0.25 120 9/20/2023
5.0.24 111 9/19/2023
5.0.23 150 9/18/2023
5.0.22 128 9/14/2023
5.0.21 121 9/13/2023
5.0.20 133 9/11/2023
5.0.19 128 9/7/2023
5.0.18 131 9/6/2023
5.0.17 125 9/5/2023
5.0.16 146 9/4/2023
5.0.15 131 9/1/2023
5.0.14 141 8/31/2023
5.0.13 153 8/30/2023
5.0.12 152 8/29/2023
5.0.11 135 8/25/2023
5.0.10 145 8/23/2023
5.0.9 134 8/18/2023
5.0.8 158 8/10/2023
5.0.7 157 8/8/2023
5.0.6 164 8/8/2023
5.0.5 178 8/7/2023
5.0.4 163 8/3/2023
5.0.3 178 7/26/2023
5.0.2 157 7/20/2023
5.0.1 167 7/14/2023
5.0.0 312 12/12/2022
4.0.5 545 6/10/2022
4.0.2 503 1/11/2022
4.0.1 459 7/19/2021
3.0.9 510 1/7/2021
3.0.7 607 9/13/2020
3.0.6 569 6/26/2020
3.0.5 547 6/26/2020
3.0.3 559 3/25/2020
3.0.2 608 3/1/2020
3.0.1 627 1/1/2020
3.0.0 598 12/23/2019
2.0.15 636 11/22/2019
2.0.14 589 11/22/2019
2.0.13 570 11/22/2019
2.0.12 569 11/21/2019
2.0.11 562 11/21/2019
2.0.10 577 11/21/2019
2.0.9 562 11/21/2019
2.0.8 735 3/3/2019
2.0.7 666 3/3/2019
2.0.6 685 3/3/2019
2.0.5 668 3/3/2019
2.0.4 717 2/7/2019
2.0.3 1,026 6/1/2018
2.0.2 1,013 5/22/2018
2.0.1 1,133 1/2/2018
1.0.12 1,001 11/2/2017
1.0.11 998 10/30/2017
1.0.10 997 10/26/2017
1.0.9 1,000 10/26/2017
1.0.8 1,027 10/26/2017
1.0.7 984 10/25/2017
1.0.6 969 10/25/2017
1.0.5 1,004 10/24/2017
1.0.4 951 10/24/2017
1.0.3 939 10/19/2017
1.0.2 1,074 10/18/2017
1.0.1 995 9/29/2017