Extracts plain text, language identification and more metadata from WARC records
☆22Apr 16, 2026Updated 7 months ago
Alternatives and similar repositories for warc2text
Users that are interested in warc2text are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Downloading all files of a language from the OSCAR (Open Super-large Crawled Aggregated coRpus)☆11Feb 28, 2023Updated 3 years ago
- Converts HTTrack crawls to WARC files☆39Aug 6, 2024Updated 2 years ago
- Library for fast text representation and classification.☆31Jan 9, 2024Updated 2 years ago
- Targetted language identifier, based on FastText and Hunspell.☆38Sep 4, 2025Updated last year
- Bicleaner is a parallel corpus classifier/cleaner that aims at detecting noisy sentence pairs in a parallel corpus.☆159Jun 18, 2024Updated 2 years ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- Docker Compose based system for running remote browsers (including Flash and Java support) connected to web archives☆16Jun 10, 2021Updated 5 years ago
- Command line tool for digging into WARC files☆52Updated this week
- English Resource Grammar☆34May 22, 2026Updated 5 months ago
- ☆12Updated this week
- ☆11Nov 21, 2025Updated 11 months ago
- Merged search-arctika and search-achon into a multi-module project☆14May 20, 2022Updated 4 years ago
- Sources is a web application that allows your team to store, manage and annotate your sources and to make them easily available to your r…☆13Jun 2, 2022Updated 4 years ago
- CDXJ Indexing of WARC/ARCs☆37Updated this week
- Open-source Chrome extension for injecting and overriding HTTP request headers☆15Jul 4, 2024Updated 2 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Utilities for working with the Unicode Script and Script Extensions properties☆11Nov 2, 2026Updated 2 weeks ago
- Automagically ignore all notifications related to work when you are on vacations☆21Aug 21, 2020Updated 6 years ago
- An append-only, key-multivalue store☆11Nov 10, 2023Updated 3 years ago
- Repository accompanying "An Open Dataset and Model for Language Identification" (Burchell et al., 2023)☆76Apr 1, 2025Updated last year
- Language identification golden dataset of tweets☆12Apr 22, 2016Updated 10 years ago
- Line shuffler for huge text file which does not fit in memory☆13Sep 7, 2026Updated 2 months ago
- This is a modified version of the AM29F016 or AM29F032 flash memory adapter board to easily DIY a Game Boy flash cartridge from J.Rodrigo…☆12Jun 20, 2022Updated 4 years ago
- a simple Elasticsearch/Dancer/Bootstrap application for demonstration☆14Sep 29, 2013Updated 13 years ago
- ArMATH: The Arabic Math Word Problem dataset. Accepted in LREC2022☆10Jun 16, 2022Updated 4 years ago
- GPU virtual machines on DigitalOcean Gradient AI • AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Tool to fix bitexts and tag near-duplicates for removal☆36Sep 4, 2025Updated last year
- Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.☆46Dec 4, 2017Updated 8 years ago
- A unified framework for attributing model behavior to model components, training data, and training dynamics.☆12Oct 25, 2026Updated 3 weeks ago
- Demo to post a tweet/tweet with image using twitter4j☆13Oct 9, 2014Updated 12 years ago
- Guide about how to watch Prime Video in HD quality on GNU/Linux☆10Sep 9, 2022Updated 4 years ago
- Freebox Homexa - L'intégration ultime des services Free☆10Nov 11, 2026Updated last week
- Google Coral containers☆13Apr 28, 2022Updated 4 years ago
- [WWW 2026] 🕸 GlotWeb: Web Indexing for Minority Languages☆17Apr 14, 2026Updated 7 months ago
- This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…☆31Aug 25, 2023Updated 3 years ago
- Serverless GPU API endpoints on Runpod - Get Bonus Credits • AdSkip the infrastructure headaches. Auto-scaling, pay-as-you-go, no-ops approach lets you focus on innovating your application.
- Tutorial on running keras model in C++ and python tensorflow☆11Oct 30, 2018Updated 8 years ago
- A framework for quick web archiving; canonical repository: https://gitea.arpa.li/JustAnotherArchivist/qwarc☆31Jan 17, 2026Updated 10 months ago
- DeDuplicator add-on for Heritrix 3☆14Oct 9, 2026Updated last month
- A simple tool for querying the Common Crawl CDX☆16Jan 10, 2026Updated 10 months ago
- [TMLR] "Can You Win Everything with Lottery Ticket?" by Tianlong Chen, Zhenyu Zhang, Jun Wu, Randy Huang, Sijia Liu, Shiyu Chang, Zhangy…☆10Sep 8, 2022Updated 4 years ago
- Command line tool to convert page layout files to the latest PAGE XML format. It supports all previous versions of the PAGE format as wel…☆25Jan 30, 2021Updated 5 years ago
- The pipeline for the OSCAR corpus☆178Nov 9, 2025Updated last year
Are these results useful?