go-crawler, v1.5.1

The library that implements crawling of all relative links for specified ones.

Handling links filtered by a custom link filter and passing a context to handlers and checkers.

crawling of all relative links for specified ones:
- use the httputils.HTTPClient interface from the github.com/thewizardplusplus/go-http-utils package;
calling of an outer handler for an each found link:
- passing a context to the crawler.LinkHandler interface;
- handling links filtered by a custom link filter (optional):
  - removing the handlers.UniqueHandler structure;
  - removing the handlers.RobotsTXTHandler structure;
custom filtering of considered links:
- passing a context to the crawler.LinkChecker interface;
- by a robots.txt file (optional):
  - use the httputils.HTTPClient interface from the github.com/thewizardplusplus/go-http-utils package.

crawling of all relative links for specified ones:
- repeated extracting of relative links on error (optional):
  - only specified repeat count;
  - supporting of delay between repeats;
- delayed extracting of relative links (optional):
  - reducing of a delay time by the time elapsed since the last request;
  - using of individual delays for each thread;
calling of an outer handler for an each found link:
- it's called directly during crawling;
- handling of links immediately after they have been extracted;
- passing of the source link in the outer handler;
- handling links filtered by a custom link filter (optional);
custom filtering of considered links:
- by relativity of a link (optional);
- by uniqueness of an extracted link (optional):
  - supporting of sanitizing of a link before checking of uniqueness (optional);
- by a robots.txt file (optional):
  - customized user agent;
- supporting of grouping of link filters:
  - result of group filtering is successful only when all filters are successful;
parallelization possibilities:
- crawling of relative links in parallel;
- supporting of background working:
  - automatic completion after processing all filtered links;
- simulate an unbounded channel of links to avoid a deadlock.

Content: code.

License: MIT.