RovoMe / JIRLbotLinks
Java implementation of the Internet Research Lab Web Crawler (IRLbot) as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in their paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
☆16Updated 8 years ago
Alternatives and similar repositories for JIRLbot
Users that are interested in JIRLbot are comparing it to the libraries listed below
Sorting:
- The LAW next generation crawler.☆89Updated 4 years ago
- API definition, resources and reference implementation of URL Frontiers☆54Updated 3 weeks ago
- Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or fi…☆197Updated this week
- A set of reusable Java components that implement functionality common to any web crawler☆250Updated last week
- A text tagger based on Lucene / Solr, using FST technology☆177Updated last year
- Java library for reading and writing WARC files with a typed API☆50Updated 2 months ago
- Query preprocessor for Java-based search engines (Querqy Core and Lucene implementation)☆189Updated last week
- Common web archive utility code.☆57Updated this week
- The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…☆72Updated last year
- Entity resolution for Elasticsearch.☆164Updated last month
- Solr Dictionary Annotator (Microservice for Spark)☆71Updated 5 years ago
- SKOS Support for Apache Lucene and Solr☆56Updated 4 years ago
- Ingest processor doing language detection for fields☆72Updated 3 years ago
- ☆12Updated 4 years ago
- Demonstration of searching PDF document with Solr, Tika, and Tesseract☆32Updated last year
- Apache Anything To Triples (Any23) is a library, a web service and a command line tool that extracts structured data in RDF format from a…☆98Updated 2 years ago
- A high performance "thin wrapper" HTTP REST server on top of Apache Lucene☆146Updated last year
- WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing,…☆112Updated 3 years ago
- Github mirror of "search/extra" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for c…☆55Updated 2 weeks ago
- command line tool for Apache Lucene☆163Updated 4 months ago
- Elasticsearch plugin for b-bit minhash algorism☆62Updated last year
- A scalable, mature and versatile web crawler based on Apache Storm☆950Updated last week
- The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)☆222Updated 2 years ago
- Examples of Solr configuration entries for Solr plugins and Conceptual Search\Semantic Search from Simon Hughes Dice.com☆26Updated 9 years ago
- Mirror of Apache Rya☆113Updated 2 years ago
- DKPro JWPL (DKPro Java Wikipedia Library) is a free, Java-based application programming interface that facilitates access to all informat…☆87Updated last week
- RDF store on a cloud-based architecture (previously on https://code.google.com/p/cumulusrdf)☆31Updated 9 years ago
- A high performance gRPC server on top of Apache Lucene☆302Updated this week
- Yet another JanusGraph, Cassandra/Scylla and Elasticsearch in Docker Compose setup☆58Updated 5 years ago
- Solr Cloud and ZooKeeper CLI☆15Updated 7 years ago