Alternatives and similar repositories for crawlers

Users that are interested in crawlers are comparing it to the libraries listed below

• crawler-commons / crawler-commons
  A set of reusable Java components that implement functionality common to any web crawler
  ☆246Updated last month
• apache / stormcrawler
  A scalable, mature and versatile web crawler based on Apache Storm
  ☆939Updated this week
• carrot2 / carrot2
  Carrot2: Text Clustering Algorithms and Applications
  ☆832Updated 2 weeks ago
• commoncrawl / commoncrawl-crawler
  The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)
  ☆222Updated 2 years ago
• USCDataScience / sparkler
  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
  ☆418Updated 2 years ago
• jatecs / jatecs
  Java text categorization system
  ☆57Updated 8 years ago
• sweble / sweble-wikitext
  The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…
  ☆73Updated last year
• flaxsearch / marple
  Browser-driven explorer for lucene indexes
  ☆74Updated 4 years ago
• apache / manifoldcf
  Mirror of Apache ManifoldCF
  ☆80Updated 2 weeks ago
• apache / opennlp-sandbox
  Apache OpenNLP Sandbox
  ☆44Updated last week
• LAW-Unimi / BUbiNG
  The LAW next generation crawler.
  ☆88Updated 3 years ago
• apache / gora
  The Apache Gora open source framework provides an in-memory data model and persistence for big data.
  ☆121Updated last year
• sonalake / utah-parser
  Java library for parsing semi-structured text files
  ☆65Updated 3 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆56Updated 3 months ago
• sematext / query-segmenter
  Solr Query Segmenter for structuring unstructured queries
  ☆22Updated 4 years ago
• labsai / EDDI
  Prompt & Conversation Management Middleware for Conversational AI APIs such as OpenAI ChatGPT, Facebook Hugging Face, Anthropic Claude, G…
  ☆280Updated last month
• gephi / graphstore
  Fast in-memory graph structure, powering Gephi
  ☆74Updated last week
• zuliaio / zuliasearch
  Zulia Search Engine
  ☆33Updated last week
• Anant / awesome-solr
  A curated list of Awesome Apache Solr links and resources.
  ☆110Updated 4 years ago
• crawljax / crawljax
  Crawljax
  ☆531Updated 2 years ago
• carrot2 / elasticsearch-carrot2
  Carrot2 plugin for ElasticSearch
  ☆291Updated 2 years ago
• DigitalPebble / TextClassification
  A Text Classification API in Java originally developed by DigitalPebble Ltd. The API is independent from the ML implementations used and …
  ☆48Updated 4 years ago
• openvenues / jpostal
  Java/JNI bindings to libpostal for for fast international street address parsing/normalization
  ☆130Updated 3 months ago
• Silverpeas / Silverpeas-Core
  Core API for Silverpeas
  ☆50Updated this week
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆55Updated last year
• hollingsworthd / ScreenSlicer
  Automatic, zero-config web scraping -- written in Java, has no dependency on Java EE or app servers, and the web scraper has a restful/JS…
  ☆155Updated 8 years ago
• Heliosearch / heliosearch
  The next generation of open source search
  ☆93Updated 8 years ago
• apache / opennlp-addons
  Mirror of Apache OpenNLP Add-ons
  ☆17Updated last week
• bejean / crawl-anywhere
  Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.
  ☆98Updated 8 years ago
• DigitalPebble / behemoth
  Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
  ☆283Updated 7 years ago