Download io.github.jasperroel JAR files with all dependencies

This project provides a simple WebCrawler with retry-capabilities, functionality to distinguish between http/https sites. It biggest feature is that it allows for plugins (or CrawlerActions), which allows you to hook your scripts into the crawling process. It also allow for setting "blocked" URLs. Those URLs or patterns will not be crawled.