go-crawler, v1.7.1

The library that implements crawling of all relative links for specified ones.

Optimizing of the sitemap.xml files processing and ignoring of the errors that occur during it.

Change Log

• crawling of all relative links for specified ones:
  • extracting links from a sitemap.xml file (optional):
    • ignoring of the error on loading of the sitemap.xml file:
      • logging of the received error;
      • returning of an empty Sitemap instead;
    • supporting of few sitemap.xml files for a single link:
      • processing of each sitemap.xml file is done in a separate goroutine;
  • supporting of grouping of link extractors:
    • extracting links concurrently:
      • processing of each link extractor is done in a separate goroutine.

Features

• crawling of all relative links for specified ones:
  • repeated extracting of relative links on error (optional):
    • only specified repeat count;
    • supporting of delay between repeats;
  • delayed extracting of relative links (optional):
    • reducing of a delay time by the time elapsed since the last request;
    • using of individual delays for each thread;
  • extracting links from a sitemap.xml file (optional):
    • ignoring of the error on loading of the sitemap.xml file:
      • logging of the received error;
      • returning of an empty Sitemap instead;
    • supporting of few sitemap.xml files for a single link:
      • processing of each sitemap.xml file is done in a separate goroutine;
    • supporting of a Sitemap index file:
      • supporting of a delay before loading of each sitemap.xml file listed in the index;
  • supporting of grouping of link extractors:
    • result of group extracting is merged results of each extractor in the group;
    • extracting links concurrently:
      • processing of each link extractor is done in a separate goroutine;
• calling of an outer handler for an each found link:
  • it's called directly during crawling;
  • handling of links immediately after they have been extracted;
  • passing of the source link in the outer handler;
  • handling links filtered by a custom link filter (optional);
  • handling links concurrently (optional);
• custom filtering of considered links:
  • by relativity of a link (optional);
  • by uniqueness of an extracted link (optional):
    • supporting of sanitizing of a link before checking of uniqueness (optional);
  • by a robots.txt file (optional):
    • customized user agent;
  • supporting of grouping of link filters:
    • result of group filtering is successful only when all filters are successful;
• parallelization possibilities:
  • crawling of relative links in parallel;
  • supporting of background working:
    • automatic completion after processing all filtered links;
  • simulate an unbounded channel of links to avoid a deadlock.

Repository

Link: https://github.com/thewizardplusplus/go-crawler/tree/v1.7.1.

Content: code.

License: MIT.