Extracts plain text, language identification and more metadata from WARC records
☆23Apr 16, 2026Updated 3 months ago
Alternatives and similar repositories for warc2text
Users that are interested in warc2text are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.☆160Jun 18, 2024Updated 2 years ago
- Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives☆16Jun 10, 2021Updated 5 years ago
- Object Resource Stream and CDXJ Drafts☆15Nov 28, 2018Updated 7 years ago
- Tool to fix bitexts and tag near-duplicates for removal☆35Sep 4, 2025Updated 10 months ago
- Arabic Transliteration in Python☆36Aug 19, 2013Updated 12 years ago
- Managed hosting for WordPress and PHP on Cloudways • AdManaged hosting for WordPress, Magento, Laravel, or PHP apps, on multiple cloud providers. Deploy in minutes on Cloudways by DigitalOcean.
- Targetted language identifier, based on FastText and Hunspell.☆38Sep 4, 2025Updated 10 months ago
- [WWW 2026] 🕸 GlotWeb: Web Indexing for Minority Languages☆17Apr 14, 2026Updated 3 months ago
- Code from Bellingcat's guide☆11Dec 8, 2022Updated 3 years ago
- Line shuffler for huge text file which does not fit in memory☆13Dec 1, 2022Updated 3 years ago
- Automagically ignore all notifications related to work when you are on vacations☆21Aug 21, 2020Updated 5 years ago
- This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…☆31Aug 25, 2023Updated 2 years ago
- Decoders for weather sensor data from RTL SDR.☆18Apr 27, 2025Updated last year
- Open-source Chrome extension for injecting and overriding HTTP request headers☆15Jul 4, 2024Updated 2 years ago
- Poor man's simple harvester for arXiv resources☆14Jul 14, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- This repository provides German documentation relating to the text recognition software Tesseract. The documentation was created in the c…☆16Sep 6, 2022Updated 3 years ago
- GC4LM: A Colossal (Biased) language model for German☆13May 2, 2021Updated 5 years ago
- Seeder - Czech webarchive curating tool and public site☆17Feb 12, 2026Updated 5 months ago
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆19Jul 24, 2025Updated 11 months ago
- Heatmap of multiclass confusion matrix☆11Sep 11, 2019Updated 6 years ago
- A simple tool for querying the Common Crawl CDX☆16Jan 10, 2026Updated 6 months ago
- A browser extension providing Open Access bibliographical services☆18Dec 9, 2022Updated 3 years ago
- ☆13Feb 9, 2022Updated 4 years ago
- MIDict (Multi-Index Dict) can be indexed by any "keys" or "values", suitable as a bidirectional/inverse dict or a multi-key/multi-value d…☆14May 19, 2016Updated 10 years ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- A Scheduler for Batched LLM Inference☆19Oct 5, 2025Updated 9 months ago
- Extension for pie to include taggers with their models and pre/postprocessors☆11Jun 23, 2026Updated 3 weeks ago
- Tool for manual evaluation of parallel sentences.☆15Jan 26, 2026Updated 5 months ago
- Myanmar and Thai Language Resources☆10Jul 18, 2022Updated 4 years ago
- ☆10Aug 26, 2021Updated 4 years ago
- Lossless normalization of uppercase characters: Go, C++ & JavaScript☆11Jul 7, 2026Updated 2 weeks ago
- ☆11Jul 23, 2023Updated 2 years ago
- A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.☆66Jul 29, 2024Updated last year
- ☆19Apr 6, 2021Updated 5 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆195Jul 9, 2025Updated last year
- A Multilingual Keyboard Layout-Based Typo Generator☆17Nov 23, 2025Updated 7 months ago
- ChatGPT solutions for the MLE interview☆14Dec 9, 2022Updated 3 years ago
- A social media open post web archiving tool☆26Feb 4, 2026Updated 5 months ago
- Servlet transport for Elasticsearch☆41Aug 8, 2024Updated last year
- Ruby Gem that makes sure that only a single instance of a code block is running.☆16Mar 13, 2013Updated 13 years ago
- Minangkabau NLP corpus. PACLIC 2020☆11Jun 7, 2021Updated 5 years ago