go-crawler, v1.11

The library that implements crawling of all relative links for specified ones.

Resolving of relative links.

Change Log

• crawling of all relative links for specified ones:
  • resolving of relative links:
    • by the base tag;
    • by the Content-Base and Content-Location headers;
    • by the request URI.

Features

• crawling of all relative links for specified ones:
  • resolving of relative links:
    • by the base tag;
    • by the Content-Base and Content-Location headers;
    • by the request URI;
  • supporting of leading and trailing spaces trimming in extracted links (optional);
  • repeated extracting of relative links on error (optional):
    • only specified repeat count;
    • supporting of delay between repeats;
  • delayed extracting of relative links (optional):
    • reducing of a delay time by the time elapsed since the last request;
    • using of individual delays for each thread;
  • extracting links from a sitemap.xml file (optional):
    • ignoring of the error on loading of the sitemap.xml file:
      • logging of the received error;
      • returning of an empty Sitemap instead;
    • supporting of few sitemap.xml files for a single link:
      • processing of each sitemap.xml file is done in a separate goroutine;
      • supporting of an outer generator for sitemap.xml links:
        • generators:
          • simple generator (it returns the sitemap.xml file in the site root);
          • hierarchical generator (it returns the suitable sitemap.xml file for each part of the URL path);
          • generator based on the robots.txt file;
        • supporting of grouping of generators:
          • result of group generating is merged results of each generator in the group;
          • generating concurrently:
            • processing of each generator is done in a separate goroutine;
    • supporting of a Sitemap index file:
      • supporting of a delay before loading of each sitemap.xml file listed in the index;
    • supporting of a gzip compression of a sitemap.xml file;
  • supporting of grouping of link extractors:
    • result of group extracting is merged results of each extractor in the group;
    • extracting links concurrently:
      • processing of each link extractor is done in a separate goroutine;
• calling of an outer handler for an each found link:
  • it's called directly during crawling;
  • handling of links immediately after they have been extracted;
  • passing of the source link in the outer handler;
  • handling links filtered by a custom link filter (optional);
  • handling links concurrently (optional);
  • supporting of grouping of outer handlers:
    • processing of each outer handler is done in a separate goroutine;
• custom filtering of considered links:
  • by relativity of a link (optional):
    • supporting of result inverting;
  • by uniqueness of an extracted link (optional):
    • supporting of sanitizing of a link before checking of uniqueness (optional);
  • by a robots.txt file (optional):
    • customized user agent;
  • supporting of grouping of link filters:
    • result of group filtering is successful only when all filters are successful;
• parallelization possibilities:
  • crawling of relative links in parallel;
  • supporting of background working:
    • automatic completion after processing all filtered links;
  • simulate an unbounded channel of links to avoid a deadlock.

Repository

Link: https://github.com/thewizardplusplus/go-crawler/tree/v1.11.

Content: code.

License: MIT.