Alternatives and similar repositories for crawlers

Users that are interested in crawlers are comparing it to the libraries listed below

• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆245Updated this week
• apache / stormcrawler
  A scalable, mature and versatile web crawler based on Apache Storm
  ☆928Updated last week
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆218Updated 2 years ago
• USCDataScience / sparkler
  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
  ☆417Updated 2 years ago
• Findwise / Hydra
  Distributed processing framework for search solutions
  ☆82Updated 2 years ago
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆122Updated last year
• apache / manifoldcf
  Mirror of Apache ManifoldCF
  ☆80Updated last month
• flaxsearch / marple
  Browser-driven explorer for lucene indexes
  ☆74Updated 3 years ago
• visallo / vertexium
  High-security graph database
  ☆65Updated 3 years ago
• LAW-Unimi / BUbiNG
  The LAW next generation crawler.
  ☆88Updated 3 years ago
• sweble / sweble-wikitext
  The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…
  ☆73Updated last year
• gephi / graphstore
  Fast in-memory graph structure, powering Gephi
  ☆75Updated last week
• carrot2 / carrot2
  Carrot2: Text Clustering Algorithms and Applications
  ☆818Updated last week
• apache / opennlp-sandbox
  Apache OpenNLP Sandbox
  ☆43Updated last week
• francelabs / datafari
  Open Source, Distributed, Big Data Enterprise Search Engine
  ☆85Updated this week
• sematext / query-segmenter
  Solr Query Segmenter for structuring unstructured queries
  ☆22Updated 4 years ago
• sematext / solr-redis
  Solr Redis Extensions
  ☆53Updated last year
• Anant / awesome-solr
  A curated list of Awesome Apache Solr links and resources.
  ☆109Updated 3 years ago
• zuliaio / zuliasearch
  Zulia Search Engine
  ☆33Updated 3 weeks ago
• bejean / crawl-anywhere
  Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.
  ☆97Updated 8 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆56Updated last month
• Norconex / importer
  Norconex Importer is a Java library and command-line application meant to "parse" and "extract" content out of a file as plain text, what…
  ☆33Updated last month
• o19s / splainer-search
  Angular JS Solr and Elasticsearch and OpenSearch Diagnostic Search Services
  ☆27Updated last month
• jatecs / jatecs
  Java text categorization system
  ☆57Updated 8 years ago
• OpenSextant / SolrTextTagger
  A text tagger based on Lucene / Solr, using FST technology
  ☆177Updated last year
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆55Updated last year
• tspannhw / nifi-extracttext-processor
  Apache NiFi Custom Processor Extracting Text From Files with Apache Tika
  ☆35Updated 2 years ago
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆283Updated 7 years ago
• datacleaner / metamodel_extras
  Extra pluggable modules for Apache MetaModel (but licensed with LGPL)
  ☆17Updated 3 years ago
• o19s / pdf-discovery-demo
  Demonstration of searching PDF document with Solr, Tika, and Tesseract
  ☆31Updated 10 months ago