commoncrawl / cc-crawl-statisticsLinks
Statistics of Common Crawl monthly archives mined from URL index files
☆189Updated last week
Alternatives and similar repositories for cc-crawl-statistics
Users that are interested in cc-crawl-statistics are comparing it to the libraries listed below
Sorting:
- Tools to construct and process Common Crawl webgraphs☆93Updated last week
- Process Common Crawl data with Python and Spark☆442Updated 3 months ago
- Index Common Crawl archives in tabular format☆124Updated 3 weeks ago
- A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine☆183Updated 7 months ago
- News crawling with StormCrawler - stores content as WARC☆353Updated 6 months ago
- Common crawl extractor☆78Updated last year
- A python utility for downloading Common Crawl data☆243Updated 2 years ago
- Various Jupyter notebooks about Common Crawl data☆55Updated 4 months ago
- A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/☆196Updated 6 years ago
- A robust web archive analytics toolkit☆114Updated 5 months ago
- Article extraction benchmark: dataset and evaluation scripts☆321Updated last year
- A dataset for pretraining language models targeted for legal tasks.☆137Updated 3 years ago
- YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training☆43Updated 4 years ago
- Streaming WARC/ARC library for fast web archive IO☆428Updated 8 months ago
- The AI Knowledge Editor☆185Updated 3 years ago
- A polite and user-friendly downloader for Common Crawl data☆51Updated last week
- Code for constructing TLDR corpus from Reddit dataset☆26Updated 3 years ago
- Fast and robust date extraction from web pages, with Python or on the command-line☆138Updated 3 weeks ago
- The pipeline for the OSCAR corpus☆171Updated last year
- Email Datasets can be found here☆69Updated 5 years ago
- Collection of Datasets for Legal Text Processing☆117Updated last month
- Download, parse, and filter data from Court Listener, part of the FreeLaw projects. Data-ready for The-Pile.☆12Updated 2 years ago
- 💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023☆148Updated 2 months ago
- Deployment of pywb as a CommonCrawl Index Server☆21Updated 7 years ago
- Python API for https://vespa.ai, the open big data serving engine☆137Updated this week
- Code for collecting, processing, and preparing datasets for the Common Pile☆221Updated this week
- ☆95Updated last year
- Official implementation of the paper "CoEdIT: Text Editing by Task-Specific Instruction Tuning" (EMNLP 2023)☆129Updated 11 months ago
- Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters☆143Updated 7 months ago
- Repository for Zheng and Guha et al., 2021, "When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Data…☆91Updated 2 years ago