peridot/vendor/github.com/gocolly/colly/v2
2022-07-07 22:13:21 +02:00
..
debug Initial commit 2022-07-07 22:13:21 +02:00
storage Initial commit 2022-07-07 22:13:21 +02:00
.codecov.yml Initial commit 2022-07-07 22:13:21 +02:00
.travis.yml Initial commit 2022-07-07 22:13:21 +02:00
BUILD.bazel Initial commit 2022-07-07 22:13:21 +02:00
CHANGELOG.md Initial commit 2022-07-07 22:13:21 +02:00
colly.go Initial commit 2022-07-07 22:13:21 +02:00
context.go Initial commit 2022-07-07 22:13:21 +02:00
CONTRIBUTING.md Initial commit 2022-07-07 22:13:21 +02:00
go.mod Initial commit 2022-07-07 22:13:21 +02:00
go.sum Initial commit 2022-07-07 22:13:21 +02:00
htmlelement.go Initial commit 2022-07-07 22:13:21 +02:00
http_backend.go Initial commit 2022-07-07 22:13:21 +02:00
http_trace.go Initial commit 2022-07-07 22:13:21 +02:00
LICENSE.txt Initial commit 2022-07-07 22:13:21 +02:00
README.md Initial commit 2022-07-07 22:13:21 +02:00
request.go Initial commit 2022-07-07 22:13:21 +02:00
response.go Initial commit 2022-07-07 22:13:21 +02:00
unmarshal.go Initial commit 2022-07-07 22:13:21 +02:00
VERSION Initial commit 2022-07-07 22:13:21 +02:00
xmlelement.go Initial commit 2022-07-07 22:13:21 +02:00

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

GoDoc Backers on Open Collective Sponsors on Open Collective build status report card view examples Code Coverage FOSSA Status Twitter URL

Features

  • Clean API
  • Fast (>1k request/sec on a single core)
  • Manages request delays and maximum concurrency per domain
  • Automatic cookie and session handling
  • Sync/async/parallel scraping
  • Caching
  • Automatic encoding of non-unicode responses
  • Robots.txt support
  • Distributed scraping
  • Configuration via environment variables
  • Extensions

Example

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("http://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

Add colly to your go.mod file:

module github.com/x/y

go 1.14

require (
        github.com/gocolly/colly/v2 latest
)

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

FOSSA Status