Alternatives and similar repositories for crawlers

Users that are interested in crawlers are comparing it to the libraries listed below

• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆251Updated this week
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆223Updated 3 years ago
• USCDataScience / sparkler
  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
  ☆419Updated 2 years ago
• LAW-Unimi / BUbiNG
  The LAW next generation crawler.
  ☆90Updated 4 years ago
• apache / stormcrawler
  A scalable, mature and versatile web crawler based on Apache Storm
  ☆956Updated this week
• apache / opennlp-sandbox
  Apache OpenNLP Sandbox
  ☆46Updated this week
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆121Updated last year
• sonalake / utah-parser
  Java library for parsing semi-structured text files
  ☆65Updated 4 years ago
• iipc / jwarc
  Java library for reading and writing WARC files with a typed API
  ☆52Updated 3 weeks ago
• visallo / vertexium
  High-security graph database
  ☆64Updated 3 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆59Updated last week
• lucidworks / zeppelin-solr
  Apache Solr interpreter for Apache Zeppelin
  ☆29Updated 2 years ago
• datacleaner / metamodel_extras
  Extra pluggable modules for Apache MetaModel (but licensed with LGPL)
  ☆17Updated 4 years ago
• sweble / sweble-wikitext
  The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…
  ☆72Updated last year
• Anant / awesome-solr
  A curated list of Awesome Apache Solr links and resources.
  ☆110Updated 4 years ago
• gephi / graphstore
  Fast in-memory graph structure, powering Gephi
  ☆79Updated this week
• apache / manifoldcf
  Mirror of Apache ManifoldCF
  ☆80Updated last week
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆283Updated 7 years ago
• crawler-commons / url-frontier
  API definition, resources and reference implementation of URL Frontiers
  ☆56Updated last month
• Findwise / Hydra
  Distributed processing framework for search solutions
  ☆82Updated 3 years ago
• ModeShape / modeshape
  ModeShape is a distributed, hierarchical, transactional, and consistent data store with support for queries, full-text search, events, ve…
  ☆221Updated 3 years ago
• flaxsearch / marple
  Browser-driven explorer for lucene indexes
  ☆74Updated 4 years ago
• sematext / solr-redis
  Solr Redis Extensions
  ☆53Updated last year
• lucidworks / storm-solr
  Storm / Solr Integration
  ☆19Updated last year
• jatecs / jatecs
  Java text categorization system
  ☆57Updated 8 years ago
• DigitalPebble / TextClassification
  A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and …
  ☆48Updated 4 years ago
• olehmberg / winter
  WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing,…
  ☆112Updated 3 years ago
• Silverpeas / Silverpeas-Core
  Core API for Silverpeas
  ☆52Updated last week
• nsoft / jesterj
  Document Ingestion Framework for Search Systems
  ☆37Updated 3 weeks ago
• carrot2 / carrot2
  Carrot2: Text Clustering Algorithms and Applications
  ☆841Updated last month