WARC (Web Archive) Input and Output Formats for Hadoop
☆38Dec 7, 2014Updated 11 years ago
Alternatives and similar repositories for warc-hadoop
Users that are interested in warc-hadoop are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- CommonCrawl Test version of Nutch☆16Jul 10, 2014Updated 12 years ago
- The Historian's WARC Toolkit☆16May 14, 2015Updated 11 years ago
- warc and wet support for Hadoop's mapreduce api☆14Mar 26, 2015Updated 11 years ago
- w3act is an annotation and curation tool for building web archive collections☆21Jan 30, 2024Updated 2 years ago
- ☆12Sep 7, 2022Updated 4 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Prototype SOLR-powered web archive exploration UI.☆44Jun 3, 2020Updated 6 years ago
- CDI and Scala Integration Case Study☆10Jun 22, 2011Updated 15 years ago
- Common Crawl fork of Apache Nutch☆44Oct 20, 2026Updated last week
- Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…☆134Nov 21, 2025Updated 11 months ago
- Warcbase is an open-source platform for managing analyzing web archives☆162Dec 8, 2017Updated 8 years ago
- Collects multimedia content shared through social networks.☆19Feb 18, 2015Updated 11 years ago
- How deep does Google Analytics go? Efficiently tackling Common Crawl using AWS & MapReduce☆17Feb 5, 2014Updated 12 years ago
- Set of Dockerfiles for localized package caching services☆10Oct 30, 2014Updated 12 years ago
- A self-contained example of summingbird running in hybrid mode☆23Mar 31, 2014Updated 12 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Rainfall is an extensible java framework to implement custom DSL based stress and performance tests☆15Mar 31, 2026Updated 7 months ago
- A set of reusable Java components that implement functionality common to any web crawler☆259Updated this week
- Storm / Solr Integration☆19Feb 2, 2024Updated 2 years ago
- ☆12Jul 14, 2014Updated 12 years ago
- A Nutch 2.2.1 plugin which allows users to shuffle off the responsibility for retrieving pages to a selenium hub/node spoke system. This …☆16Jun 9, 2016Updated 10 years ago
- Naive Bayes classifier written in Scalding☆27Feb 23, 2014Updated 12 years ago
- UPenn Talk about Julia and ML☆14Nov 14, 2014Updated 11 years ago
- A Python library to simplify batch requests to AWS Services☆12Apr 25, 2020Updated 6 years ago
- es-nozzle synchronizes directories into ElasticSearch☆14Jan 22, 2014Updated 12 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Common Crawl support library to access 2008-2012 crawl archives (ARC files)☆507Nov 29, 2017Updated 8 years ago
- A half-day workshop on Scalding, the Scala API for Cascading☆12Mar 25, 2016Updated 10 years ago
- This is a TREC evaluation demonstration written for a lecture on information retrieval evaluation.☆24Feb 12, 2018Updated 8 years ago
- Common Crawl One-Oh-One (aka "A Common Crawl Experiment")☆26Oct 31, 2014Updated 12 years ago
- IPython Notebook for Sentiment Classification☆10Nov 12, 2014Updated 11 years ago
- TREC Core track☆11Jul 5, 2017Updated 9 years ago
- Disciplined Convex Programming in R using Convex.jl.☆14Dec 18, 2018Updated 7 years ago
- Example CAPI implementation.☆10Jan 2, 2021Updated 5 years ago
- Arteria is a high performance message channel system for IPC and network communication☆12Jun 21, 2017Updated 9 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- ☆50Feb 22, 2017Updated 9 years ago
- Saves proxied HTTP traffic to a WARC file.☆27Oct 22, 2013Updated 13 years ago
- An experimental Java bytecode to C++ transpiler☆13Jan 24, 2016Updated 10 years ago
- Python script to parse the netvizz output.☆11Feb 4, 2014Updated 12 years ago
- ☆25Apr 6, 2015Updated 11 years ago
- luajit version for xxHash☆10Apr 28, 2019Updated 7 years ago
- Reproducing Distributed Systems and Experiments on Cloud☆41Sep 11, 2023Updated 3 years ago