A small web crawler named aranea (Latin for spider). https://www.bananas-playground.net/projekt/aranea/

Banana 18c49ef61f callback does work differently		преди 1 седмица
documentation	32e5d2b1d9 transactions	преди 1 седмица
lib	7271145682 license change and new config value	преди 1 месец
storage	24fb355861 fetch.pl	преди 2 години
.gitignore	546a78d2ab ignore config file	преди 1 седмица
CHANGELOG	36b6e40cf8 avoid big downloads	преди 1 седмица
COPYING	7271145682 license change and new config value	преди 1 месец
LICENSE	7271145682 license change and new config value	преди 1 месец
README.md	5800264126 better readme	преди 1 седмица
TODO	36b6e40cf8 avoid big downloads	преди 1 седмица
VERSION	17aef3b5ab cleanup of the code and some paperwork	преди 2 години
cleanup.pl	32e5d2b1d9 transactions	преди 1 седмица
config.default.txt	36b6e40cf8 avoid big downloads	преди 1 седмица
fetch.pl	18c49ef61f callback does work differently	преди 1 седмица
parse-results.pl	32e5d2b1d9 transactions	преди 1 седмица
setup.sql	cfdca6000e project cleanup and updated project website links	преди 2 години

aranea

A small web crawler named aranea (Latin for spider). The aim is to gather unique domains to show what is out there.

Fetch

It starts with a given set of URL(s) and parses them for more URLs. Stores them and fetches them too. perl fetch.pl

Each URL result (Stored result from the call) will be parsed for other URLs to follow. perl parse-results.pl

After a run cleanup will gather all the unique Domains into a table. Removes URLs from the fetch table which are already enough. perl cleanup.pl

The table url_to_ignore does have a small amount of domains and part of domains which will be ignored. Adding a global SPAM list would be overkill.

A good idea is to run it with a DNS filter, which has a good blocklist.