Alternatives and similar repositories for crawlers:

Users that are interested in crawlers are comparing it to the libraries listed below

• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆244Updated 3 weeks ago
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆213Updated 2 years ago
• Norconex / importer
  Norconex Importer is a Java library and command-line application meant to "parse" and "extract" content out of a file as plain text, what…
  ☆34Updated 6 months ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆55Updated last month
• Smerity / cc-warc-examples
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆56Updated 4 years ago
• visallo / vertexium
  High-security graph database
  ☆62Updated 2 years ago
• lucidworks / silk
  Silk is a port of Kibana 4 project.
  ☆70Updated 8 years ago
• orientechnologies / orientdb-gremlin
  TinkerPop3 Graph Structure Implementation for OrientDB
  ☆94Updated last week
• iipc / jwarc
  Java library for reading and writing WARC files with a typed API
  ☆48Updated 4 months ago
• spinscale / elasticsearch-opennlp-plugin
  Additional opennlp mapping type for elasticsearch in order to perform named entity recognition
  ☆136Updated 8 years ago
• LAW-Unimi / BUbiNG
  The LAW next generation crawler.
  ☆87Updated 3 years ago
• gephi / graphstore
  Fast in-memory graph structure, powering Gephi
  ☆75Updated 5 months ago
• sematext / query-segmenter
  Solr Query Segmenter for structuring unstructured queries
  ☆21Updated 3 years ago
• OpenSextant / SolrTextTagger
  A text tagger based on Lucene / Solr, using FST technology
  ☆176Updated last year
• Syncleus / Ferma
  An ORM / OGM for the TinkerPop graph stack.
  ☆137Updated 2 years ago
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆121Updated last year
• carrot2 / elasticsearch-carrot2
  Carrot2 plugin for ElasticSearch
  ☆291Updated 2 years ago
• USCDataScience / sparkler
  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
  ☆415Updated 2 years ago
• flaxsearch / marple
  Browser-driven explorer for lucene indexes
  ☆74Updated 3 years ago
• Wolfgang-Schuetzelhofer / jcypher
  Java access to Neo4J graph databases at multiple levels of abstraction
  ☆85Updated 4 years ago
• francelabs / datafari
  Open Source, Distributed, Big Data Enterprise Search Engine
  ☆69Updated last month
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆281Updated 7 years ago
• openvenues / jpostal
  Java/JNI bindings to libpostal for for fast international street address parsing/normalization
  ☆116Updated last year
• USCDataScience / SentimentAnalysisParser
  Combines Apache OpenNLP and Apache Tika and provides facilities for automatically deriving sentiment from text.
  ☆34Updated last year
• momer / nutch-selenium
  ☆28Updated 8 years ago
• neo4j-contrib / neo4j-jdbc-2x
  Neo4j JDBC driver
  ☆69Updated last year
• fengtan / sophie
  A Solr browser and administration tool
  ☆27Updated 4 years ago
• Norconex / collector-filesystem
  Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to netw…
  ☆22Updated 7 months ago
• invesdwin / invesdwin-webproxy
  A parallel download manager for web scraping that supports proxy servers
  ☆9Updated 3 weeks ago
• jprante / elasticsearch-skywalker
  Skywalker for Elasticsearch is like Luke for Lucene
  ☆79Updated 5 years ago