warc and wet support for Hadoop's mapreduce api
☆14Mar 26, 2015Updated 11 years ago
Alternatives and similar repositories for warc-mapreduce
Users that are interested in warc-mapreduce are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Miscellaneous tools for processing WARC files from the CommonCrawl☆25Jan 1, 2014Updated 12 years ago
- ☆23Feb 22, 2024Updated 2 years ago
- Common web archive utility code.☆66Updated this week
- CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop☆56Apr 26, 2021Updated 5 years ago
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆54Jun 12, 2020Updated 6 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- How deep does Google Analytics go? Efficiently tackling Common Crawl using AWS & MapReduce☆17Feb 5, 2014Updated 12 years ago
- builds the PubChem Bioassay database that is distributed for use with bioassayR☆11Dec 4, 2021Updated 4 years ago
- Events and Situations Ontology☆15Apr 20, 2018Updated 8 years ago
- WARC (Web Archive) Input and Output Formats for Hadoop☆38Dec 7, 2014Updated 11 years ago
- ggtikz: Add TikZ annotations with absolute data or relative plot coordinates to ggplots☆11Nov 6, 2025Updated last year
- Source code and data files for "PEMT: A patent enrichment tool with applicability in drug discovery"☆17Sep 28, 2026Updated last month
- ☆15Jul 5, 2022Updated 4 years ago
- ☆12Jul 14, 2014Updated 12 years ago
- A proof-of-concept in-browser JavaScript-based search engine☆12Aug 25, 2015Updated 11 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- VisNAB☆12Apr 5, 2013Updated 13 years ago
- Python script to create CDX index files of WARC data☆22Sep 4, 2025Updated last year
- A half-day workshop on Scalding, the Scala API for Cascading☆12Mar 25, 2016Updated 10 years ago
- A Hadoop job that runs GATE applications☆15Oct 16, 2013Updated 13 years ago
- Stand alone utilities for Scribe log server (Scribe C++ clients, Scribe C++ shared library)☆11Jun 8, 2012Updated 14 years ago
- Create HTML timelines by exporting .org files☆11Mar 30, 2026Updated 7 months ago
- [CVPR 25] MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures☆16Mar 22, 2026Updated 7 months ago
- Python package developed to evaluate textual overlap (N-Grams) between two volumes of text.☆11Sep 23, 2021Updated 5 years ago
- ☆16May 10, 2023Updated 3 years ago
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- Python library for reading and writing warc files☆251Mar 7, 2022Updated 4 years ago
- ☆15Jan 3, 2015Updated 11 years ago
- Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…☆134Nov 21, 2025Updated 11 months ago
- family finance tools of a closet librarian☆13Updated this week
- Flax-maintained fork of Lucene/Solr with support for interval queries☆15Oct 9, 2015Updated 11 years ago
- A collection of utilities and tools for the JACK audio ecosystem☆18Nov 25, 2023Updated 2 years ago
- Set of scripts to aid in the download of the GDELT data files from gdelt.utdallas.edu☆18May 14, 2014Updated 12 years ago
- ☆12Jun 10, 2016Updated 10 years ago
- JSON Schema files for Pandoc JSON☆14Aug 19, 2014Updated 12 years ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Nondestructive warc-in-tar to warc conversion☆27Apr 21, 2013Updated 13 years ago
- Nice Opaque Identifier☆16Sep 21, 2023Updated 3 years ago
- Cycling, Incremental, Compressed, Encrypted Backups to Amazon S3.☆15Jun 7, 2021Updated 5 years ago
- Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)☆178Aug 18, 2025Updated last year
- Launch AWS Elastic MapReduce jobs that process Common Crawl data.☆49Feb 15, 2017Updated 9 years ago
- The core SolrSherlock project, which consists of many sub-projects☆16Sep 8, 2015Updated 11 years ago
- Scan a folder of document files of all types and extract the text into a CSV suitable for Overview☆26Mar 23, 2016Updated 10 years ago