RovoMe / JIRLbotLinks
Java implementation of the Internet Research Lab Web Crawler (IRLbot) as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in their paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
☆16Updated 8 years ago
Alternatives and similar repositories for JIRLbot
Users that are interested in JIRLbot are comparing it to the libraries listed below
Sorting:
- The LAW next generation crawler.☆90Updated 4 years ago
- API definition, resources and reference implementation of URL Frontiers☆55Updated 2 weeks ago
- Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or fi…☆197Updated this week
- A set of reusable Java components that implement functionality common to any web crawler☆251Updated last week
- CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop☆56Updated 4 years ago
- Java library for reading and writing WARC files with a typed API☆52Updated this week
- Ingest processor doing language detection for fields☆72Updated 3 years ago
- The Common Crawl Crawler Engine and Related MapReduce code (2008-2012)☆223Updated 3 years ago
- Github mirror of "search/highlighter" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access…☆104Updated last month
- Common web archive utility code.☆57Updated 3 weeks ago
- Entity resolution for Elasticsearch.☆164Updated last week
- A scalable, mature and versatile web crawler based on Apache Storm☆953Updated this week
- A text tagger based on Lucene / Solr, using FST technology☆177Updated 2 years ago
- Github mirror of "search/extra" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for c…☆55Updated last month
- Common Crawl fork of Apache Nutch☆40Updated last week
- An elasticsearch plugin to create hierarchical aggregations☆52Updated last month
- A high performance "thin wrapper" HTTP REST server on top of Apache Lucene☆146Updated last year
- command line tool for Apache Lucene☆163Updated this week
- Index Common Crawl archives in tabular format☆124Updated this week
- SOLR bulk indexing utility for the command line.☆45Updated last month
- A curated list of Awesome Apache Solr links and resources.☆110Updated 4 years ago
- Apache Anything To Triples (Any23) is a library, a web service and a command line tool that extracts structured data in RDF format from a…☆98Updated 2 years ago
- The Sweble Wikitext Components module provides a parser for MediaWiki's wikitext and an engine trying to emulate the behavior of a MediaW…☆72Updated last year
- A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/☆205Updated 7 years ago
- A bundle of useful Elasticsearch plugins☆112Updated last year
- An Elasticsearch ingest processor to do named entity extraction using Apache OpenNLP☆275Updated 3 years ago
- Angular JS Solr and Elasticsearch and OpenSearch Diagnostic Search Services☆27Updated 2 months ago
- DKPro JWPL (DKPro Java Wikipedia Library) is a free, Java-based application programming interface that facilitates access to all informat…☆87Updated this week
- Highly performant, lightweight framework for linked data processing. Supports RDFa, JSON-LD, RDF/XML and plain text formats, runs on Andr…☆50Updated 3 years ago
- WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing,…☆112Updated 3 years ago