gzipstream allows Python to process multi-part gzip files from a streaming source
☆23Feb 24, 2017Updated 9 years ago
Alternatives and similar repositories for gzipstream
Users that are interested in gzipstream are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Deployment of pywb as a CommonCrawl Index Server☆22Oct 6, 2017Updated 9 years ago
- Machine assisted dossiers☆19Oct 12, 2017Updated 9 years ago
- A python library to deal with scientific papers.☆17Apr 2, 2016Updated 10 years ago
- A space for code and projects around analysing news content☆23Feb 8, 2018Updated 8 years ago
- carebot-tracker.js — Carebot's tracking component for Google Analytics events☆17Apr 19, 2016Updated 10 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Universalizing Open-Access Journals & Papers☆19Mar 8, 2017Updated 9 years ago
- A simple ElasticSearch plugin wrapping around the search endpoint to provide Rocchio query expansion☆19Aug 5, 2017Updated 9 years ago
- Turning news into events since 2014.☆52May 1, 2017Updated 9 years ago
- Find rss, atom, xml, and rdf feeds on webpages☆31Sep 16, 2026Updated last month
- Extract text information from warc files☆12Sep 1, 2022Updated 4 years ago
- ☆11May 9, 2021Updated 5 years ago
- Extraction and Interface for Oxford English Dictionary (OED) 1st Edition☆23Aug 29, 2013Updated 13 years ago
- A Python port of the Perl address parser available at http://search.cpan.org/~timb/Geo-StreetAddress-US-1.03/US.pm☆26Jan 31, 2019Updated 7 years ago
- crdts on ipfs☆11Dec 2, 2015Updated 10 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆46Dec 4, 2017Updated 8 years ago
- Common web archive utility code.☆66Oct 1, 2026Updated last month
- ☆21Feb 1, 2018Updated 8 years ago
- The code reproduces the results of the experiments in the paper. In particular, it performs experiments in which machine-learning models …☆21Aug 16, 2021Updated 5 years ago
- Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends☆57Jan 28, 2024Updated 2 years ago
- Change hash syntax☆10Jul 16, 2012Updated 14 years ago
- For the filthiest web scrapers that have no time for rate-limits.☆20Oct 11, 2020Updated 6 years ago
- Process Common Crawl data with Python and Spark☆459Sep 17, 2026Updated last month
- DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…☆54Jun 12, 2020Updated 6 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- Rust Extensions for the Ruby gem case_transform☆10Jul 27, 2017Updated 9 years ago
- Anytime Ranking for Impact-Ordered Indexes☆13Jan 9, 2017Updated 9 years ago
- Hierarchical Greedy Graph☆16Jul 3, 2022Updated 4 years ago
- The Firmware for the Freifunk Bielefeld Community☆22Oct 17, 2020Updated 6 years ago
- classify a job description (or noisy job title) into a ONET job title☆19Oct 14, 2016Updated 10 years ago
- Paginating the web☆36Feb 11, 2014Updated 12 years ago
- A simple command line interface to Google's search index.☆12Jun 18, 2010Updated 16 years ago
- Streaming WARC/ARC library for fast web archive IO☆484Updated this week
- A rust crate to create & manage cluster☆11Oct 22, 2026Updated last week
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Haircut - the URL shortener with Padrino and MongoDB☆10Feb 14, 2011Updated 15 years ago
- Strongly typed javascript objects, self-validating, detailed error reports☆12Aug 13, 2020Updated 6 years ago
- WARC (Web Archive) Input and Output Formats for Hadoop☆38Dec 7, 2014Updated 11 years ago
- CoCrawler is a versatile web crawler built using modern tools and concurrency.☆192Apr 29, 2022Updated 4 years ago
- Lightweight and extensible Angular implementation of the D3 donut graph.☆11Nov 30, 2014Updated 11 years ago
- Higher Order Functions for Ruby☆10Apr 1, 2026Updated 7 months ago
- Python 3 library for reading and writing warc files☆21Jan 29, 2018Updated 8 years ago