go-crawler, v1.4

The library that implements crawling of all relative links for specified ones.

Delayed extracting of relative links and refactoring.

Change Log

• crawling of all relative links for specified ones:
  • delayed extracting of relative links (optional):
    • reducing of a delay time by the time elapsed since the last request;
    • using of individual delays for each thread;
• refactoring:
  • extracting utility entities for syncing to the single package;
  • passing a thread ID to an extractor;
  • extractors.RepeatingExtractor structure:
    • passing an abstract sleeper as a parameter;
    • checking of call count in tests.

Features

• crawling of all relative links for specified ones:
  • repeated extracting of relative links on error (optional):
    • only specified repeat count;
    • supporting of delay between repeats;
  • delayed extracting of relative links (optional):
    • reducing of a delay time by the time elapsed since the last request;
    • using of individual delays for each thread;
• calling of an outer handler for an each found link:
  • it's called directly during crawling;
  • handling of links immediately after they have been extracted;
  • passing of the source link in the outer handler;
  • handling only of unique links (optional):
    • supporting of sanitizing of a link before checking of uniqueness (optional);
• custom filtering of considered links:
  • by relativity of a link (optional);
  • by uniqueness of an extracted link (optional):
    • supporting of sanitizing of a link before checking of uniqueness (optional);
  • supporting of grouping of link filters:
    • result of group filtering is successful only when all filters are successful;
• parallelization possibilities:
  • crawling of relative links in parallel;
  • supporting of background working:
    • automatic completion after processing all filtered links;
  • simulate an unbounded channel of links to avoid a deadlock.

Examples

crawler.Crawl():

package main

import (
    "context"
    "fmt"
    "html/template"
    stdlog "log"
    "net/http"
    "net/http/httptest"
    "os"
    "runtime"
    "strings"
    "sync"
    "time"

    "github.com/go-log/log/print"
    crawler "github.com/thewizardplusplus/go-crawler"
    "github.com/thewizardplusplus/go-crawler/checkers"
    "github.com/thewizardplusplus/go-crawler/extractors"
    "github.com/thewizardplusplus/go-crawler/handlers"
    "github.com/thewizardplusplus/go-crawler/register"
    "github.com/thewizardplusplus/go-crawler/sanitizing"
    htmlselector "github.com/thewizardplusplus/go-html-selector"
)

type LinkHandler struct {
    ServerURL string
}

func (handler LinkHandler) HandleLink(sourceLink string, link string) {
    fmt.Printf(
        "have got the link %q from the page %q\n",
        handler.replaceServerURL(link),
        handler.replaceServerURL(sourceLink),
    )
}

// replace the test server URL for reproducibility of the example
func (handler LinkHandler) replaceServerURL(link string) string {
    return strings.Replace(link, handler.ServerURL, "http://example.com", -1)
}

func RunServer() *httptest.Server {
    return httptest.NewServer(http.HandlerFunc(func(
        writer http.ResponseWriter,
        request *http.Request,
    ) {
        var links []string
        switch request.URL.Path {
        case "/":
            links = []string{"/1", "/2", "/2", "https://golang.org/"}
        case "/1":
            links = []string{"/1/1", "/1/2"}
        case "/2":
            links = []string{"/2/1", "/2/2"}
        }
        for index := range links {
            if strings.HasPrefix(links[index], "/") {
                links[index] = "http://" + request.Host + links[index]
            }
        }

        template, _ := template.New("").Parse( // nolint: errcheck
            `<ul>
                {{ range $link := . }}
                    <li><a href="{{ $link }}">{{ $link }}</a></li>
                {{ end }}
            </ul>`,
        )
        template.Execute(writer, links) // nolint: errcheck
    }))
}

func main() {
    server := RunServer()
    defer server.Close()

    logger := stdlog.New(os.Stderr, "", stdlog.LstdFlags|stdlog.Lmicroseconds)
    // wrap the standard logger via the github.com/go-log/log package
    wrappedLogger := print.New(logger)

    crawler.Crawl(
        context.Background(),
        runtime.NumCPU(),
        1000,
        []string{server.URL},
        crawler.CrawlDependencies{
            LinkExtractor: extractors.RepeatingExtractor{
                LinkExtractor: extractors.NewDelayingExtractor(
                    time.Second,
                    time.Sleep,
                    extractors.DefaultExtractor{
                        HTTPClient: http.DefaultClient,
                        Filters: htmlselector.OptimizeFilters(htmlselector.FilterGroup{
                            "a": {"href"},
                        }),
                    },
                ),
                RepeatCount:  5,
                RepeatDelay:  0,
                Logger:       wrappedLogger,
                SleepHandler: time.Sleep,
            },
            LinkChecker: checkers.CheckerGroup{
                checkers.HostChecker{
                    Logger: wrappedLogger,
                },
                checkers.DuplicateChecker{
                    LinkRegister: register.NewLinkRegister(
                        sanitizing.SanitizeLink,
                        wrappedLogger,
                    ),
                },
            },
            LinkHandler: handlers.UniqueHandler{
                // don't use here the link register from the duplicate checker above
                LinkRegister: register.NewLinkRegister(
                    sanitizing.SanitizeLink,
                    wrappedLogger,
                ),
                LinkHandler: LinkHandler{
                    ServerURL: server.URL,
                },
            },
            Logger: wrappedLogger,
        },
    )

    // Unordered output:
    // have got the link "http://example.com/1" from the page "http://example.com"
    // have got the link "http://example.com/1/1" from the page "http://example.com/1"
    // have got the link "http://example.com/1/2" from the page "http://example.com/1"
    // have got the link "http://example.com/2" from the page "http://example.com"
    // have got the link "http://example.com/2/1" from the page "http://example.com/2"
    // have got the link "http://example.com/2/2" from the page "http://example.com/2"
    // have got the link "https://golang.org/" from the page "http://example.com"
}

crawler.HandleLinksConcurrently() with delaying extracting:

package main

import (
    "context"
    "fmt"
    "html/template"
    stdlog "log"
    "net/http"
    "net/http/httptest"
    "os"
    "runtime"
    "strings"
    "sync"
    "time"

    "github.com/go-log/log/print"
    crawler "github.com/thewizardplusplus/go-crawler"
    "github.com/thewizardplusplus/go-crawler/checkers"
    "github.com/thewizardplusplus/go-crawler/extractors"
    "github.com/thewizardplusplus/go-crawler/handlers"
    "github.com/thewizardplusplus/go-crawler/register"
    "github.com/thewizardplusplus/go-crawler/sanitizing"
    htmlselector "github.com/thewizardplusplus/go-html-selector"
)

type LinkHandler struct {
    ServerURL string
}

func (handler LinkHandler) HandleLink(sourceLink string, link string) {
    fmt.Printf(
        "have got the link %q from the page %q\n",
        handler.replaceServerURL(link),
        handler.replaceServerURL(sourceLink),
    )
}

// replace the test server URL for reproducibility of the example
func (handler LinkHandler) replaceServerURL(link string) string {
    return strings.Replace(link, handler.ServerURL, "http://example.com", -1)
}

func RunServer() *httptest.Server {
    return httptest.NewServer(http.HandlerFunc(func(
        writer http.ResponseWriter,
        request *http.Request,
    ) {
        var links []string
        switch request.URL.Path {
        case "/":
            links = []string{"/1", "/2", "/2", "https://golang.org/"}
        case "/1":
            links = []string{"/1/1", "/1/2"}
        case "/2":
            links = []string{"/2/1", "/2/2"}
        }
        for index := range links {
            if strings.HasPrefix(links[index], "/") {
                links[index] = "http://" + request.Host + links[index]
            }
        }

        template, _ := template.New("").Parse( // nolint: errcheck
            `<ul>
                {{ range $link := . }}
                    <li><a href="{{ $link }}">{{ $link }}</a></li>
                {{ end }}
            </ul>`,
        )
        template.Execute(writer, links) // nolint: errcheck
    }))
}

func main() {
    server := RunServer()
    defer server.Close()

    links := make(chan string, 1000)
    links <- server.URL

    var waiter sync.WaitGroup
    waiter.Add(1)

    logger := stdlog.New(os.Stderr, "", stdlog.LstdFlags|stdlog.Lmicroseconds)
    // wrap the standard logger via the github.com/go-log/log package
    wrappedLogger := print.New(logger)

    crawler.HandleLinksConcurrently(
        context.Background(),
        runtime.NumCPU(),
        links,
        crawler.HandleLinkDependencies{
            CrawlDependencies: crawler.CrawlDependencies{
                LinkExtractor: extractors.RepeatingExtractor{
                    LinkExtractor: extractors.NewDelayingExtractor(
                        time.Second,
                        time.Sleep,
                        extractors.DefaultExtractor{
                            HTTPClient: http.DefaultClient,
                            Filters: htmlselector.OptimizeFilters(htmlselector.FilterGroup{
                                "a": {"href"},
                            }),
                        },
                    ),
                    RepeatCount:  5,
                    RepeatDelay:  0,
                    Logger:       wrappedLogger,
                    SleepHandler: time.Sleep,
                },
                LinkChecker: checkers.CheckerGroup{
                    checkers.HostChecker{
                        Logger: wrappedLogger,
                    },
                    checkers.DuplicateChecker{
                        LinkRegister: register.NewLinkRegister(
                            sanitizing.SanitizeLink,
                            wrappedLogger,
                        ),
                    },
                },
                LinkHandler: handlers.UniqueHandler{
                    // don't use here the link register from the duplicate checker above
                    LinkRegister: register.NewLinkRegister(
                        sanitizing.SanitizeLink,
                        wrappedLogger,
                    ),
                    LinkHandler: LinkHandler{
                        ServerURL: server.URL,
                    },
                },
                Logger: wrappedLogger,
            },
            Waiter: &waiter,
        },
    )

    waiter.Wait()

    // Unordered output:
    // have got the link "http://example.com/1" from the page "http://example.com"
    // have got the link "http://example.com/1/1" from the page "http://example.com/1"
    // have got the link "http://example.com/1/2" from the page "http://example.com/1"
    // have got the link "http://example.com/2" from the page "http://example.com"
    // have got the link "http://example.com/2/1" from the page "http://example.com/2"
    // have got the link "http://example.com/2/2" from the page "http://example.com/2"
    // have got the link "https://golang.org/" from the page "http://example.com"
}

Repository

Link: https://github.com/thewizardplusplus/go-crawler/tree/v1.4.

Content: code.

License: MIT.