The robots.txt exclusion protocol implementation for Go language
☆284Feb 19, 2026Updated last week
Alternatives and similar repositories for robotstxt
Users that are interested in robotstxt are comparing it to the libraries listed below
Sorting:
- tiny Go library to normalize URLs☆483Mar 19, 2024Updated last year
- grobotstxt is a native Go port of Google's robots.txt parser and matcher library.☆114Mar 16, 2022Updated 3 years ago
- A simple and flexible web crawler that follows the robots.txt policies and crawl delays.☆790May 19, 2021Updated 4 years ago
- Polite, slim and concurrent web crawler.☆2,052May 19, 2021Updated 4 years ago
- Better `go run`. Build go codes transparently and exec☆16Oct 14, 2018Updated 7 years ago
- Generate Go structs from JSON!☆42May 1, 2013Updated 12 years ago
- Mutation testing for Go☆27Mar 17, 2013Updated 12 years ago
- Loom - A repository for Weave Files☆26Jun 12, 2015Updated 10 years ago
- Timeout invocation. Go porting of GNU timeout and able to use as Go package☆75Apr 21, 2019Updated 6 years ago
- A little like that j-thing, only in Go.☆14,918Updated this week
- Get extra stat (creation time, access time) from os.FileInfo for golang☆14May 19, 2021Updated 4 years ago
- ☆25Apr 30, 2015Updated 10 years ago
- (GitHub) Pkg Critic is a tool combining search results from GoDoc API and stats info from GitHub API.☆15Mar 14, 2016Updated 9 years ago
- Go package to minify HTML (alpha-quality)☆15Mar 30, 2024Updated last year
- A GO web application profiler.☆59Apr 6, 2018Updated 7 years ago
- WebLoop: Scriptable, headless WebKit with a Go API. Like PhantomJS, but for Go.☆1,365Apr 1, 2024Updated last year
- CSS selector library in Go☆752Dec 19, 2024Updated last year
- gofind searches through Go source code by types.☆48Jul 13, 2023Updated 2 years ago
- go-querystring is Go library for encoding structs into URL query strings.☆2,132Updated this week
- A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.☆3,073May 2, 2023Updated 2 years ago
- Golang pkg for URL parsing and normalization☆162Jun 22, 2022Updated 3 years ago
- Html Content / Article Extractor in Golang☆450Aug 3, 2025Updated 7 months ago
- Minimal cgo bindings for libenca☆19Mar 15, 2016Updated 9 years ago
- A Go package concerning quoted-printable encoding.☆20Jul 16, 2015Updated 10 years ago
- Verify IP addresses of respectful crawlers like Googlebot by reverse dns and forward dns lookups☆36Feb 20, 2026Updated last week
- A Go library for performing Unicode Text Segmentation as described in Unicode Standard Annex #29☆88Dec 19, 2022Updated 3 years ago
- A Go language server.☆33Mar 25, 2019Updated 6 years ago
- A Go implementation of Amazon Ion Hash.☆23Nov 20, 2025Updated 3 months ago
- Hatena Notation (はてな記法) Parser written in Go☆17Aug 3, 2017Updated 8 years ago
- Kallax is a PostgreSQL typesafe ORM for the Go language.☆855Dec 28, 2022Updated 3 years ago
- Randomized testing for Go☆4,846Sep 24, 2024Updated last year
- A web crawler / scraper engine written in Golang☆30May 9, 2016Updated 9 years ago
- gta: do transitive analysis to find packages whose dependencies have changed☆224Feb 13, 2026Updated 2 weeks ago
- Crash your app in style (Golang)☆3,716Mar 13, 2025Updated 11 months ago
- Implementation of "An Optimal Suffix Array Construction Algorithm" described in a Technical Report by Ge Nong☆26Feb 26, 2013Updated 13 years ago
- (Deprecated) Elasticsearch command line tool (for Elasticsearch 1.x)☆28Feb 27, 2017Updated 9 years ago
- A CC-BY logo to use for your Perl projects☆21Jul 22, 2025Updated 7 months ago
- Common juju errors and functions to annotate errors. Based on juju/errgo☆1,394Oct 23, 2023Updated 2 years ago
- Imaging is a simple image processing package for Go☆5,677Sep 21, 2023Updated 2 years ago