Alternatives and similar repositories for cc-downloader:

Users that are interested in cc-downloader are comparing it to the libraries listed below

• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆11Updated 7 months ago
• commoncrawl / web-languages
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …
  ☆38Updated this week
• explosion / wikid
  Generate a SQLite database from Wikipedia & Wikidata dumps.
  ☆33Updated last year
• bigscience-workshop / lam
  Libraries, Archives and Museums (LAM)
  ☆82Updated 2 years ago
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆21Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process webgraphs from Common Crawl data
  ☆87Updated this week
• r-three / common-pile
  Repo to hold code and track issues for the collection of permissively licensed data
  ☆23Updated this week
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• bitextor / warc2text
  Extracts plain text, language identification and more metadata from WARC records
  ☆21Updated 3 weeks ago
• karlb / sqlite-spellfix
  Loadable spellfix1 extension for sqlite as python package
  ☆26Updated 11 months ago
• iipc / warcaroo
  ☆12Updated 3 months ago
• explosion / spacy-vectors-builder
  🌸 Train floret vectors
  ☆18Updated last year
• Pleias / marginalia
  ☆67Updated last year
• anjackson / sliver
  A tool for collection archival slivers of the web and web archives
  ☆13Updated last month
• internetarchive / arch
  Web application for distributed compute analysis of Archive-It web archive collections.
  ☆16Updated 2 weeks ago
• cophi-wue / pydelta
  an experimental implementation of Burrow's delta in Python 3
  ☆21Updated 3 years ago
• internetarchive / fatcat-scholar
  search interface for scholarly works
  ☆84Updated 8 months ago
• pjox / gutf
  Terminal tool that converts files encoding to UTF-8
  ☆10Updated 5 years ago
• Pleias / pleias_ScholasticAI
  ☆50Updated last month
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆175Updated 2 weeks ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• MarcosFP97 / textscope
  A text analysis library for relevance and subtheme detection
  ☆16Updated last month
• JustlyAI / lmss_entity_extractor
  Tool to apply Legal Matter Specification Standard (LMSS) to documents
  ☆13Updated 7 months ago
• nrv / pimmi
  Python IMage MIning
  ☆13Updated 2 weeks ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 4 months ago
• fbkarsdorp / tmi
  Flask Interface to Thompson's Motif Index
  ☆18Updated 5 years ago
• steffenfritz / filedriller
  ☆14Updated last year
• wjbmattingly / hobbit-spacy
  ☆23Updated last year
• nlnwa / warchaeology
  Command line tool for digging into WARC files
  ☆39Updated this week
• mozilla / distilvit
  image-to-text model for PDF.js
  ☆36Updated 2 weeks ago