A small web crawler named aranea (Latin for spider). https://www.bananas-playground.net/projekt/aranea/
Banana
18c49ef61f
callback does work differently
|
1 هفته پیش | |
|---|---|---|
| documentation | 1 هفته پیش | |
| lib | 1 ماه پیش | |
| storage | 2 سال پیش | |
| .gitignore | 1 هفته پیش | |
| CHANGELOG | 1 هفته پیش | |
| COPYING | 1 ماه پیش | |
| LICENSE | 1 ماه پیش | |
| README.md | 1 هفته پیش | |
| TODO | 1 هفته پیش | |
| VERSION | 2 سال پیش | |
| cleanup.pl | 1 هفته پیش | |
| config.default.txt | 1 هفته پیش | |
| fetch.pl | 1 هفته پیش | |
| parse-results.pl | 1 هفته پیش | |
| setup.sql | 2 سال پیش |
README.md
aranea
https://www.bananas-playground.net/projekt/aranea
A small web crawler named aranea (Latin for spider). The aim is to gather unique domains to show what is out there.
Fetch
It starts with a given set of URL(s) and parses them for more
URLs. Stores them and fetches them too. perl fetch.pl
Parse
Each URL result (Stored result from the call) will be parsed
for other URLs to follow. perl parse-results.pl
Cleanup
After a run cleanup will gather all the unique Domains into
a table. Removes URLs from the fetch table which are already
enough. perl cleanup.pl
Ignores
The table url_to_ignore does have a small amount of domains and part of domains which will be ignored.
Adding a global SPAM list would be overkill.
A good idea is to run it with a DNS filter, which has a good blocklist.