RovoMe / JIRLbotLinks
Java implementation of the Internet Research Lab Web Crawler (IRLbot) as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in their paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
☆16Updated 8 years ago
Alternatives and similar repositories for JIRLbot
Users that are interested in JIRLbot are comparing it to the libraries listed below
Sorting:
- The LAW next generation crawler.☆90Updated 4 years ago
- API definition, resources and reference implementation of URL Frontiers☆56Updated last month
- A set of reusable Java components that implement functionality common to any web crawler☆251Updated this week
- Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or fi…☆196Updated 2 weeks ago
- Entity resolution for Elasticsearch.☆165Updated last month
- The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…☆72Updated last year
- Common web archive utility code.☆59Updated 2 weeks ago
- Lightning fast spell correction / fuzzy search library based on SymSpell by Commerce-Experts☆81Updated 7 years ago
- A text tagger based on Lucene / Solr, using FST technology☆177Updated 2 years ago
- Query preprocessor for Java-based search engines (Querqy Core and Lucene implementation)☆189Updated last month
- SOLR bulk indexing utility for the command line.☆45Updated 2 months ago
- A scalable, mature and versatile web crawler based on Apache Storm☆956Updated this week
- Apache NLPCraft - API to convert natural language into actions.☆83Updated 7 months ago
- Angular JS Solr and Elasticsearch and OpenSearch Diagnostic Search Services☆28Updated last week
- Carrot2: Text Clustering Algorithms and Applications☆841Updated last month
- Java imap nio client that is designed to scale well for thousands of connections per machine and reduce contention when using large numbe…☆62Updated 11 months ago
- ☆12Updated 4 years ago
- The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)☆223Updated 3 years ago
- DKPro JWPL (DKPro Java Wikipedia Library) is a free, Java-based application programming interface that facilitates access to all informat…☆87Updated this week
- Ingest processor doing language detection for fields☆72Updated 3 years ago
- Implementation of Vision Based Page Segmentation algorithm in Java☆104Updated 6 years ago
- Demonstration of searching PDF document with Solr, Tika, and Tesseract☆32Updated last year
- Solr Dictionary Annotator (Microservice for Spark)☆71Updated 5 years ago
- Browser-driven explorer for lucene indexes☆74Updated 4 years ago
- Github mirror of "search/highlighter" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access…☆104Updated last month
- Github mirror of "search/extra" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for c…☆55Updated last month
- Apache Solr interpreter for Apache Zeppelin☆29Updated 2 years ago
- Java library for reading and writing WARC files with a typed API☆52Updated 3 weeks ago
- Lucene Directory implementation for AWS S3☆45Updated 3 weeks ago
- A high performance "thin wrapper" HTTP REST server on top of Apache Lucene☆146Updated last year