A small web crawler named aranea (Latin for spider). https://www.bananas-playground.net/projekt/aranea/

Banana 738bae354b typos and code cleanup 1 week ago
documentation cafc554c1b documentation 1 week ago
lib cfdca6000e project cleanup and updated project website links 1 week ago
storage 24fb355861 fetch.pl 1 week ago
CHANGELOG 17aef3b5ab cleanup of the code and some paperwork 4 months ago
LICENSE 17aef3b5ab cleanup of the code and some paperwork 4 months ago
README cfdca6000e project cleanup and updated project website links 1 week ago
VERSION 17aef3b5ab cleanup of the code and some paperwork 4 months ago
cleanup.pl 738bae354b typos and code cleanup 1 week ago
config.txt cfdca6000e project cleanup and updated project website links 1 week ago
fetch.pl 738bae354b typos and code cleanup 1 week ago
parse-results.pl cfdca6000e project cleanup and updated project website links 1 week ago
setup.sql cfdca6000e project cleanup and updated project website links 1 week ago

README

https://://www.bananas-playground.net/projekt/aranea

A small web crawler named aranea (Latin for spider).
The aim is to gather unique domains to show what is out there.

It starts with a given set of URL(s) and parses them for more
URLs. Stores them and fethches them too.
-> fetch.pl

Each URL result (Stored result from the call) will be parsed
for other URLs to follow.
-> parse.pl

After a run cleanup will gather all the uniqe Domains into
a table. Removes URLs from the fetch table which are already
enough.
-> cleanup.pl