Alternatives and similar repositories for nutch:

Users that are interested in nutch are comparing it to the libraries listed below

• iipc / webarchive-commons
  Common web archive utility code.
  ☆52Updated last month
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆109Updated 7 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆50Updated 4 years ago
• khundman / marve
  For extracting measurements and related entities from text
  ☆57Updated 4 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆110Updated 2 months ago
• idio / json-wikipedia
  Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump. Questions? https://gitter.im/idio-opensource/Lobby
  ☆17Updated 2 years ago
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆194Updated 7 years ago
• DiceTechJobs / VectorsInSearch
  Dice.com repo to accompany the dice.com 'Vectors in Search' talk by Simon Hughes, from the Activate 2018 search conference, and the 'Sear…
  ☆85Updated 3 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆42Updated 5 years ago
• DiceTechJobs / RelevancyTuning
  Dice.com tutorial on using black box optimization algorithms to do relevancy tuning on your Solr Search Engine Configuration from Simon H…
  ☆28Updated 5 years ago
• Smerity / cc-warc-examples
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆56Updated 3 years ago
• adbar / py3langid
  Faster, modernized fork of the language identification tool langid.py
  ☆50Updated 2 months ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆43Updated 7 years ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆148Updated 4 months ago
• filipecasal / knowledge-repo
  ☆16Updated 3 years ago
• marcusklang / wikiforia
  A Utility Library for Wikipedia dumps
  ☆33Updated 7 years ago
• archivesunleashed / aut
  The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
  ☆140Updated 11 months ago
• LanguageMachines / ucto
  Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic pr…
  ☆68Updated this week
• kermitt2 / grobid-ner
  A Named-Entity Recogniser based on Grobid.
  ☆50Updated 4 months ago
• xiaoganghan / wikientities
  Linking Entities in CommonCrawl Dataset onto Wikipedia Concepts
  ☆59Updated 12 years ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆121Updated 5 months ago
• trec-kba / streamcorpus
  common data interchange format for document processing pipelines that apply natural language processing tools to large streams of text
  ☆34Updated 8 years ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆395Updated last month
• USCDataScience / autoextractor
  A toolkit for clustering web pages based on various similarity measures.
  ☆33Updated 3 years ago
• EuropeanaNewspapers / ner-corpora
  Named Entity Recognition data for Europeana Newspapers
  ☆171Updated last year
• tballison / quaerite
  Search relevance evaluation toolkit
  ☆31Updated 2 years ago
• knowitall / nlptools
  A toolkit that wraps various natural language processing implementations behind a common interface.
  ☆101Updated 7 years ago
• opener-project / ner-base
  NER tagger for English, Spanish, Dutch, Italian and German and French.
  ☆35Updated 9 years ago
• USCDataScience / NLTKRest
  This is a REST Server endpoint built using Flask and Python.
  ☆24Updated 2 years ago