RovoMe / JIRLbotLinks
Java implementation of the Internet Research Lab Web Crawler (IRLbot) as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in their paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
☆16Updated 8 years ago
Alternatives and similar repositories for JIRLbot
Users that are interested in JIRLbot are comparing it to the libraries listed below
Sorting:
- The LAW next generation crawler.☆88Updated 3 years ago
- A set of reusable Java components that implement functionality common to any web crawler☆246Updated 3 weeks ago
- API definition, resources and reference implementation of URL Frontiers☆52Updated 2 weeks ago
- Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or fi…☆194Updated 2 weeks ago
- Ingest processor doing language detection for fields☆72Updated 2 years ago
- The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)☆221Updated 2 years ago
- A high performance "thin wrapper" HTTP REST server on top of Apache Lucene☆145Updated last year
- A high performance gRPC server on top of Apache Lucene☆298Updated this week
- Querqy for Elasticsearch☆47Updated last week
- Various utility scripts for running Lucene performance tests☆218Updated this week
- Github mirror of "search/extra" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for c…☆55Updated last month
- Query preprocessor for Java-based search engines (Querqy Core and Lucene implementation)☆187Updated last week
- Zulia Search Engine☆33Updated last week
- command line tool for Apache Lucene☆163Updated 3 months ago
- Apache NLPCraft - API to convert natural language into actions.☆83Updated 4 months ago
- Java imap nio client that is designed to scale well for thousands of connections per machine and reduce contention when using large numbe…☆60Updated 8 months ago
- Elasticsearch plugin for b-bit minhash algorism☆62Updated last year
- A text tagger based on Lucene / Solr, using FST technology☆177Updated last year
- Solr Dictionary Annotator (Microservice for Spark)☆71Updated 5 years ago
- Benchmark of open source, embedded, memory-mapped, key-value stores available from Java (JMH)☆142Updated 2 years ago
- The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…☆73Updated last year
- Browser-driven explorer for lucene indexes☆74Updated 4 years ago
- Let's try batching some cypher queries☆11Updated 9 years ago
- PATRICIA, Double Array, LOUDS Trie implementations for Java☆180Updated last year
- Asynchronous search makes it possible for users to run queries in the background, allowing users to track the progress, and retrieve par…☆23Updated 4 years ago
- Various utilities regarding Levenshtein transducers. (Java)☆58Updated 3 years ago
- DKPro JWPL (DKPro Java Wikipedia Library) is a free, Java-based application programming interface that facilitates access to all informat…☆86Updated last week
- Provides a SQL interface to your TinkerPop enabled graph db☆75Updated 2 years ago
- Entity resolution for Elasticsearch.☆162Updated last week
- Match tens of thousands of regular expressions within milliseconds - Java bindings for Intel's hyperscan 5☆193Updated 2 weeks ago