Alternatives and similar repositories for nutch

Users that are interested in nutch are comparing it to the libraries listed below

• commoncrawl / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆196Updated this week
• commoncrawl / news-crawl
  News crawling with StormCrawler - stores content as WARC
  ☆360Updated 11 months ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 5 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆124Updated 3 weeks ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆205Updated 7 years ago
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 8 years ago
• commoncrawl / cc-pyspark
  Process Common Crawl data with Python and Spark
  ☆452Updated last week
• dalab / web2text
  Source code for the paper "Web2Text: Deep Structured Boilerplate Removal", full paper @ ECIR'18
  ☆170Updated 4 years ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆193Updated 3 years ago
• dbpedia / table-extractor
  Extract Data from Wikipedia Tables
  ☆34Updated 8 years ago
• scrapinghub / webstruct
  NER toolkit for HTML data
  ☆259Updated last year
• iipc / webarchive-commons
  Common web archive utility code.
  ☆59Updated 2 weeks ago
• filipecasal / knowledge-repo
  ☆15Updated 4 years ago
• dragnet-org / dragnet_data
  code and data used to build a training dataset for dragnet models
  ☆10Updated 5 years ago
• terrier-org / terrier-core
  Terrier IR Platform
  ☆270Updated last month
• opensemanticsearch / solr-ontology-tagger
  Automatic tagging and analysis of documents in an Apache Solr index for faceted search by RDF(S) Ontologies & SKOS thesauri
  ☆47Updated 4 years ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆442Updated last year
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆156Updated 3 months ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆58Updated last year
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆198Updated 8 years ago
• Tpt / freebase-browser
  ☆12Updated 4 years ago
• DiceTechJobs / VectorsInSearch
  Dice.com repo to accompany the dice.com 'Vectors in Search' talk by Simon Hughes, from the Activate 2018 search conference, and the 'Sear…
  ☆86Updated 4 years ago
• openeventdata / es-geonames
  Create a Geonames gazetteer index in Elasticsearch
  ☆79Updated 2 years ago
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆223Updated 3 years ago
• lintool / warcbase
  Warcbase is an open-source platform for managing analyzing web archives
  ☆162Updated 8 years ago
• OpenSextant / SolrTextTagger
  A text tagger based on Lucene / Solr, using FST technology
  ☆177Updated 2 years ago
• internetarchive / umbra
  A queue-controlled browser automation tool for improving web crawl quality
  ☆64Updated 5 months ago
• opensemanticsearch / open-semantic-search-apps
  Python/Django based webapps and web user interfaces for search, structure (meta data management like thesaurus, ontologies, annotations a…
  ☆99Updated 3 years ago
• dbpedia-spotlight / wikipedia-stats-extractor
  Raw Wikipedia counts for entity linking
  ☆19Updated 8 years ago
• idio / json-wikipedia
  Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump. Questions? https://gitter.im/idio-opensource/Lobby
  ☆17Updated 3 years ago