Alternatives and similar repositories for nutch

Users that are interested in nutch are comparing it to the libraries listed below

• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 8 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆122Updated last month
• commoncrawl / news-crawl
  News crawling with StormCrawler - stores content as WARC
  ☆355Updated 7 months ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆199Updated 6 years ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆430Updated 9 months ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 3 years ago
• commoncrawl / cc-pyspark
  Process Common Crawl data with Python and Spark
  ☆440Updated this week
• filipecasal / knowledge-repo
  ☆16Updated 4 years ago
• commoncrawl / cc-mrjob
  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework
  ☆167Updated 3 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆183Updated 8 months ago
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆220Updated 2 years ago
• Parsely / serpextract
  Easy extraction of keywords and engines from search engine results pages (SERPs).
  ☆91Updated 3 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 5 years ago
• seomoz / simhash-py
  Simhash and near-duplicate detection
  ☆419Updated 2 years ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆322Updated last year
• mikemccand / chromium-compact-language-detector
  Automatically exported from code.google.com/p/chromium-compact-language-detector
  ☆161Updated 4 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆56Updated last month
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• scrapinghub / webstruct
  NER toolkit for HTML data
  ☆259Updated last year
• dalab / web2text
  Source code for the paper "Web2Text: Deep Structured Boilerplate Removal", full paper @ ECIR'18
  ☆170Updated 3 years ago
• GregBowyer / cld2-cffi
  Python bindings to the Compact Language Detector
  ☆33Updated 5 years ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆62Updated last month
• spinscale / elasticsearch-opennlp-plugin
  Additional opennlp mapping type for elasticsearch in order to perform named entity recognition
  ☆136Updated 9 years ago
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆195Updated 8 years ago
• jpbruinsslot / warc3
  Python 3 library for reading and writing warc files
  ☆21Updated 7 years ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆140Updated last month
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 6 years ago
• seomoz / simhash-db-py
  Python API for Various DB-Backed Simhash Clusters
  ☆64Updated 8 years ago
• internetarchive / warc
  Python library for reading and writing warc files
  ☆244Updated 3 years ago
• Smerity / cc-warc-examples
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆57Updated 4 years ago