go-crawler, v1.10

The library that implements crawling of all relative links for specified ones.

Supporting of leading and trailing spaces trimming in extracted links and supporting of grouping of outer handlers.

Change Log

• crawling of all relative links for specified ones:
  • supporting of leading and trailing spaces trimming in extracted links (optional);
• calling of an outer handler for an each found link:
  • supporting of grouping of outer handlers:
    • processing of each outer handler is done in a separate goroutine;
• custom filtering of considered links:
  • by relativity of a link (optional):
    • supporting of result inverting;
• extend the logging:
  • in the crawler.HandleLink() function;
  • in the extractors package:
    • in the RepeatingExtractor structure;
    • in the SitemapExtractor structure;
  • in the checkers package:
    • in the HostChecker structure;
    • in the RobotsTXTChecker structure;
  • in the registers.LinkRegister structure;
• examples:
  • fix the output messages;
  • add the example with few handlers.

Features

• crawling of all relative links for specified ones:
  • supporting of leading and trailing spaces trimming in extracted links (optional);
  • repeated extracting of relative links on error (optional):
    • only specified repeat count;
    • supporting of delay between repeats;
  • delayed extracting of relative links (optional):
    • reducing of a delay time by the time elapsed since the last request;
    • using of individual delays for each thread;
  • extracting links from a sitemap.xml file (optional):
    • ignoring of the error on loading of the sitemap.xml file:
      • logging of the received error;
      • returning of an empty Sitemap instead;
    • supporting of few sitemap.xml files for a single link:
      • processing of each sitemap.xml file is done in a separate goroutine;
      • supporting of an outer generator for sitemap.xml links:
        • generators:
          • simple generator (it returns the sitemap.xml file in the site root);
          • hierarchical generator (it returns the suitable sitemap.xml file for each part of the URL path);
          • generator based on the robots.txt file;
        • supporting of grouping of generators:
          • result of group generating is merged results of each generator in the group;
          • generating concurrently:
            • processing of each generator is done in a separate goroutine;
    • supporting of a Sitemap index file:
      • supporting of a delay before loading of each sitemap.xml file listed in the index;
    • supporting of a gzip compression of a sitemap.xml file;
  • supporting of grouping of link extractors:
    • result of group extracting is merged results of each extractor in the group;
    • extracting links concurrently:
      • processing of each link extractor is done in a separate goroutine;
• calling of an outer handler for an each found link:
  • it's called directly during crawling;
  • handling of links immediately after they have been extracted;
  • passing of the source link in the outer handler;
  • handling links filtered by a custom link filter (optional);
  • handling links concurrently (optional);
  • supporting of grouping of outer handlers:
    • processing of each outer handler is done in a separate goroutine;
• custom filtering of considered links:
  • by relativity of a link (optional):
    • supporting of result inverting;
  • by uniqueness of an extracted link (optional):
    • supporting of sanitizing of a link before checking of uniqueness (optional);
  • by a robots.txt file (optional):
    • customized user agent;
  • supporting of grouping of link filters:
    • result of group filtering is successful only when all filters are successful;
• parallelization possibilities:
  • crawling of relative links in parallel;
  • supporting of background working:
    • automatic completion after processing all filtered links;
  • simulate an unbounded channel of links to avoid a deadlock.

Examples

crawler.Crawl() with few handlers:

package main

import (
    "context"
    "fmt"
    "html/template"
    stdlog "log"
    "net/http"
    "net/http/httptest"
    "os"
    "runtime"
    "strings"
    "time"

    "github.com/go-log/log/print"
    crawler "github.com/thewizardplusplus/go-crawler"
    "github.com/thewizardplusplus/go-crawler/checkers"
    "github.com/thewizardplusplus/go-crawler/extractors"
    "github.com/thewizardplusplus/go-crawler/handlers"
    "github.com/thewizardplusplus/go-crawler/models"
    urlutils "github.com/thewizardplusplus/go-crawler/url-utils"
    htmlselector "github.com/thewizardplusplus/go-html-selector"
)

type LinkHandler struct {
    Name      string
    ServerURL string
}

func (handler LinkHandler) HandleLink(
    ctx context.Context,
    link models.SourcedLink,
) {
    fmt.Printf(
        "[%s] received link %q from page %q\n",
        handler.Name,
        handler.replaceServerURL(link.Link),
        handler.replaceServerURL(link.SourceLink),
    )
}

// replace the test server URL for reproducibility of the example
func (handler LinkHandler) replaceServerURL(link string) string {
    return strings.Replace(link, handler.ServerURL, "http://example.com", -1)
}

func RunServer() *httptest.Server {
    return httptest.NewServer(http.HandlerFunc(func(
        writer http.ResponseWriter,
        request *http.Request,
    ) {
        var links []string
        switch request.URL.Path {
        case "/":
            links = []string{"/1", "/2", "/2", "https://golang.org/"}
        case "/1":
            links = []string{"/1/1", "/1/2"}
        case "/2":
            links = []string{"/2/1", "/2/2"}
        }
        for index := range links {
            if strings.HasPrefix(links[index], "/") {
                links[index] = "http://" + request.Host + links[index]
            }
        }

        template, _ := template.New("").Parse( // nolint: errcheck
            `<ul>
                {{ range $link := . }}
                    <li><a href="{{ $link }}">{{ $link }}</a></li>
                {{ end }}
            </ul>`,
        )
        template.Execute(writer, links) // nolint: errcheck
    }))
}

func main() {
    server := RunServer()
    defer server.Close()

    logger := stdlog.New(os.Stderr, "", stdlog.LstdFlags|stdlog.Lmicroseconds)
    // wrap the standard logger via the github.com/go-log/log package
    wrappedLogger := print.New(logger)

    crawler.Crawl(
        context.Background(),
        crawler.ConcurrencyConfig{
            ConcurrencyFactor: runtime.NumCPU(),
            BufferSize:        1000,
        },
        []string{server.URL},
        crawler.CrawlDependencies{
            LinkExtractor: extractors.RepeatingExtractor{
                LinkExtractor: extractors.DefaultExtractor{
                    TrimLink:   urlutils.TrimLink,
                    HTTPClient: http.DefaultClient,
                    Filters: htmlselector.OptimizeFilters(htmlselector.FilterGroup{
                        "a": {"href"},
                    }),
                },
                RepeatCount:  5,
                RepeatDelay:  time.Second,
                Logger:       wrappedLogger,
                SleepHandler: time.Sleep,
            },
            LinkChecker: checkers.HostChecker{
                ComparisonResult: urlutils.Same,
                Logger:           wrappedLogger,
            },
            LinkHandler: handlers.HandlerGroup{
                handlers.CheckedHandler{
                    LinkChecker: checkers.HostChecker{
                        ComparisonResult: urlutils.Same,
                        Logger:           wrappedLogger,
                    },
                    LinkHandler: LinkHandler{
                        Name:      "inner",
                        ServerURL: server.URL,
                    },
                },
                handlers.CheckedHandler{
                    LinkChecker: checkers.HostChecker{
                        ComparisonResult: urlutils.Different,
                        Logger:           wrappedLogger,
                    },
                    LinkHandler: LinkHandler{
                        Name:      "outer",
                        ServerURL: server.URL,
                    },
                },
            },
            Logger: wrappedLogger,
        },
    )

    // Unordered output:
    // [inner] received link "http://example.com/1" from page "http://example.com"
    // [inner] received link "http://example.com/1/1" from page "http://example.com/1"
    // [inner] received link "http://example.com/1/2" from page "http://example.com/1"
    // [inner] received link "http://example.com/2" from page "http://example.com"
    // [inner] received link "http://example.com/2" from page "http://example.com"
    // [inner] received link "http://example.com/2/1" from page "http://example.com/2"
    // [inner] received link "http://example.com/2/1" from page "http://example.com/2"
    // [inner] received link "http://example.com/2/2" from page "http://example.com/2"
    // [inner] received link "http://example.com/2/2" from page "http://example.com/2"
    // [outer] received link "https://golang.org/" from page "http://example.com"
}

Repository

Link: https://github.com/thewizardplusplus/go-crawler/tree/v1.10.

Content: code.

License: MIT.