go-crawler, v1.10.1

The library that implements crawling of all relative links for specified ones.

Perform the refactoring: add the extractors.TrimmingExtractor structure and ignore errors from each extractor in the group, instead of logging them.

Change Log

• perform the refactoring:
  • link trimming:
    • add the extractors.TrimmingExtractor structure;
    • remove the link trimming from the extractors.DefaultExtractor structure;
  • fix the extractors.ExtractorGroup structure:
    • ignore errors from each extractor in the group, instead of logging them;
  • add the registers.BasicRegister structure:
    • use in the registers.RobotsTXTRegister structure;
    • use in the registers.SitemapRegister structure;
• fix the bugs:
  • fix the tests of the checkers.HostChecker.CheckLink() method.

Repository

Link: https://github.com/thewizardplusplus/go-crawler/tree/v1.10.1.

Content: code.

License: MIT.