A small web crawler named aranea (Latin for spider). https://www.bananas-playground.net/projekt/aranea/
Banana
cafc554c1b
documentation
|
há 2 anos atrás | |
|---|---|---|
| documentation | há 2 anos atrás | |
| lib | há 2 anos atrás | |
| storage | há 2 anos atrás | |
| CHANGELOG | há 2 anos atrás | |
| LICENSE | há 2 anos atrás | |
| README | há 2 anos atrás | |
| VERSION | há 2 anos atrás | |
| cleanup.pl | há 2 anos atrás | |
| config.txt | há 2 anos atrás | |
| fetch.pl | há 2 anos atrás | |
| parse-results.pl | há 2 anos atrás | |
| setup.sql | há 2 anos atrás |
README
https://://www.bananas-playground.net/projekt/aranea
A small web crawler named aranea (Latin for spider).
The aim is to gather unique domains to show what is out there.
It starts with a given set of URL(s) and parses them for more
URLs. Stores them and fethches them too.
-> fetch.pl
Each URL result (Stored result from the call) will be parsed
for other URLs to follow.
-> parse.pl
After a run cleanup will gather all the uniqe Domains into
a table. Removes URLs from the fetch table which are already
enough.
-> cleanup.pl
A small web crawler named aranea (Latin for spider).
The aim is to gather unique domains to show what is out there.
It starts with a given set of URL(s) and parses them for more
URLs. Stores them and fethches them too.
-> fetch.pl
Each URL result (Stored result from the call) will be parsed
for other URLs to follow.
-> parse.pl
After a run cleanup will gather all the uniqe Domains into
a table. Removes URLs from the fetch table which are already
enough.
-> cleanup.pl