Alternatives and similar repositories for crawlers:

Users that are interested in crawlers are comparing it to the libraries listed below

• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆240Updated last month
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆212Updated 2 years ago
• apache / incubator-stormcrawler
  A scalable, mature and versatile web crawler based on Apache Storm
  ☆898Updated this week
• Norconex / importer
  Norconex Importer is a Java library and command-line application meant to "parse" and "extract" content out of a file as plain text, what…
  ☆33Updated 3 months ago
• Norconex / collector-filesystem
  Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to netw…
  ☆22Updated 4 months ago
• jaeksoft / opensearchserver
  Open-source Enterprise Grade Search Engine Software
  ☆503Updated 2 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆54Updated last year
• iipc / webarchive-commons
  Common web archive utility code.
  ☆52Updated last month
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆120Updated 11 months ago
• LAW-Unimi / BUbiNG
  The LAW next generation crawler.
  ☆87Updated 3 years ago
• Smerity / cc-warc-examples
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆56Updated 3 years ago
• basis-technology-corp / Java-readability
  A port of the arclabs 'readability' package to Java
  ☆72Updated 12 years ago
• sematext / query-segmenter
  Solr Query Segmenter for structuring unstructured queries
  ☆21Updated 3 years ago
• bejean / crawl-anywhere
  Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.
  ☆96Updated 7 years ago
• OpenSextant / SolrTextTagger
  A text tagger based on Lucene / Solr, using FST technology
  ☆176Updated last year
• flaxsearch / marple
  Browser-driven explorer for lucene indexes
  ☆74Updated 3 years ago
• zelandiya / maui
  ☆184Updated 6 years ago
• commoncrawl / cc-warc-examples
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
  ☆37Updated last month
• RichardHightower / slumberdb
  The JSON database for REST and Websocket storage
  ☆42Updated 10 years ago
• ohmdb / ohmdb
  OhmDB - The Irresistible RDBMS + NoSQL Database for Java
  ☆63Updated 9 years ago
• carrot2 / carrot2
  Carrot2: Text Clustering Algorithms and Applications
  ☆793Updated 3 months ago
• jatecs / jatecs
  Java text categorization system
  ☆55Updated 7 years ago
• healthonnet / hon-lucene-synonyms
  Solr query parser plugin that performs proper query-time synonym expansion.
  ☆150Updated 3 years ago
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆281Updated 6 years ago
• fivesmallq / web-data-extractor
  Extracting and parsing structured data with jQuery Selector, XPath or JsonPath from common web format like HTML, XML and JSON.
  ☆54Updated last year
• Findwise / Hydra
  Distributed processing framework for search solutions
  ☆81Updated 2 years ago
• o19s / splainer-search
  Angular JS Solr and Elasticsearch and OpenSearch Diagnostic Search Services
  ☆26Updated 7 months ago
• commoncrawl / example-warc-java
  ☆48Updated 7 years ago
• mohaps / xtractor
  XTractor is an algorithmic text extractor from web pages written in Java. It builds upon the "commonly used web design practices" approac…
  ☆43Updated 8 years ago
• ept / warc-hadoop
  WARC (Web Archive) Input and Output Formats for Hadoop
  ☆35Updated 10 years ago