rossf7 / elasticrawl
Launch AWS Elastic MapReduce jobs that process Common Crawl data.
☆49Updated 8 years ago
Alternatives and similar repositories for elasticrawl:
Users that are interested in elasticrawl are comparing it to the libraries listed below
- Human-Powered Data Analysis with Mechanical Turk☆300Updated 12 years ago
- Hadoop jobs for WikiReverse project. Parses Common Crawl data for links to Wikipedia articles.☆38Updated 6 years ago
- Index URLs in Common Crawl☆193Updated 7 years ago
- The Summarizer from the Web IR / NLP Group (WING), hence SWING, is a modular, state-of-the-art automatic extractive text summarization sy…☆39Updated 10 years ago
- Demonstration of using Python to process the Common Crawl dataset with the mrjob framework☆166Updated 2 years ago
- Parser and standardizer for politician, individual and organization names.☆129Updated 7 years ago
- RDF-Centric Map/Reduce Framework and Freebase data conversion tool☆148Updated 3 years ago
- CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop☆56Updated 3 years ago
- Extract postal addresses from the DOM☆66Updated 12 years ago
- Semanticizest: dump parser and client☆20Updated 8 years ago
- Elasticsearch Latent Semantic Indexing experimentation☆33Updated 5 years ago
- See https://github.com/tworavens/tworavens for current repository for this project and http://2ra.vn for project pages.☆30Updated 6 years ago
- ☆29Updated 8 years ago
- SKOS analysis for Elasticsearch☆54Updated 8 years ago
- Wikipedia Live Monitor☆21Updated 3 months ago
- Apache Nutch fork tunned for web services and data discovery.☆9Updated 9 years ago
- Supervised learning for novelty detection in text☆78Updated 8 years ago
- A collection and conversion of WARN notices from California☆12Updated 8 years ago
- Keeps a mirror of DBpedia live in sync☆26Updated 3 years ago
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆51Updated 4 years ago
- ☆21Updated 6 years ago
- Ruby client library for controlling Google Refine☆44Updated 6 years ago
- Curiosity is a generic frontend for facetting, displaying and editing data from any elasticsearch index.☆76Updated 9 years ago
- KEA 5.0 (keyphrase extraction software), modified to be an XML-RPC service☆42Updated 13 years ago
- Updates to Zope's keyphrase extractor (forked from 1.1.0)☆66Updated 7 years ago
- ImageCat is an Apache OODT RADIX application that uses Apache Solr, Apache Tika and Apache OODT to ingest 10s of millions of files (image…☆95Updated 6 years ago
- A lightweight server to allow HTTP requests to the Stanford Named Entity Recognized and a heavily modified CLAVIN geoparser.☆119Updated 2 years ago
- Quickly analyze and explore email with advanced analytics and visualization.☆56Updated 3 years ago
- Raw Wikipedia counts for entity linking☆19Updated 7 years ago
- Faceted search engine for domain-specific exploration of the Web☆45Updated 8 years ago