Alternatives and similar repositories for cc-downloader:

Users that are interested in cc-downloader are comparing it to the libraries listed below

• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆14Updated 8 months ago
• commoncrawl / web-languages
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …
  ☆39Updated last week
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• bigscience-workshop / lam
  Libraries, Archives and Museums (LAM)
  ☆82Updated 2 years ago
• nlnwa / warchaeology
  Command line tool for digging into WARC files
  ☆39Updated 3 weeks ago
• internetarchive / fatcat-scholar
  search interface for scholarly works
  ☆85Updated 8 months ago
• Pleias / marginalia
  ☆67Updated last year
• internetarchive / arch
  Web application for distributed compute analysis of Archive-It web archive collections.
  ☆18Updated last month
• bitextor / warc2text
  Extracts plain text, language identification and more metadata from WARC records
  ☆21Updated last month
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆21Updated last year
• r-three / common-pile
  Repo to hold code and track issues for the collection of permissively licensed data
  ☆23Updated 2 weeks ago
• cophi-wue / pydelta
  an experimental implementation of Burrow's delta in Python 3
  ☆21Updated 3 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• anjackson / sliver
  A tool for collection archival slivers of the web and web archives
  ☆13Updated 2 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆177Updated 2 weeks ago
• lfoppiano / PhD-Thesis
  PhD Dissertation "Automated Extraction and Curation of Materials Information from Scientific Literature"
  ☆9Updated last year
• archivesunleashed / notebooks
  Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archive…
  ☆26Updated 2 years ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 4 months ago
• AngryLoki / mreid-resolver
  Tool for showing Freebase and Google Knowledge Graph entries
  ☆19Updated last year
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆25Updated 5 months ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆123Updated last month
• tedunderwood / fictional-time-with-GPT4
  An experiment replicating part of "Why Literary Time is Measured in Minutes" with GPT-4.
  ☆32Updated 2 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆169Updated 3 months ago
• davanstrien / flyswot
  Command Line Interface for running 🤗 Transformers Image Classification locally
  ☆19Updated 2 weeks ago
• adbar / py3langid
  Faster, modernized fork of the language identification tool langid.py
  ☆55Updated 5 months ago
• harvard-lil / wacz-exhibitor
  Experimental proxy and wrapper for safely embedding Web Archives (warc, warc.gz, wacz) into web pages.
  ☆31Updated last month
• internetarchive / surt
  Sort-friendly URI Reordering Transform (SURT) python module
  ☆42Updated 8 months ago
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆90Updated 3 weeks ago
• iipc / warc-specifications
  Centralised repository for WARC usage specifications.
  ☆110Updated 5 months ago
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆25Updated 4 months ago