Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆71Updated last week
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆57Updated 2 months ago
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 3 years ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆194Updated last week
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆185Updated last week
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆36Updated 3 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆131Updated last week
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 3 years ago
• pdf-association / pdf-corpora
  An index of PDF-centric corpora
  ☆142Updated 3 months ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆66Updated this week
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆35Updated this week
• Volifter / Belligcat-Hackathon
  ☆21Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆99Updated last week
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 8 years ago
• Lookyloo / PlaywrightCapture
  Capture a URL with Playwright
  ☆30Updated 2 weeks ago
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆19Updated last month
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 8 years ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆70Updated 8 months ago
• ukwa / webarchive-discovery
  Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in t…
  ☆130Updated 3 months ago
• mikahama / natas
  Python 3 library for processing historical English
  ☆67Updated last year
• ail-project / Similarius
  Similarius is a Python library to compare web page and evaluate the level of similarity.
  ☆21Updated last month
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆17Updated last year
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆153Updated 3 weeks ago
• freelawproject / x-ray
  A tool to detect whether a PDF has a bad redaction
  ☆152Updated 3 weeks ago
• o19s / pdf-discovery-demo
  Demonstration of searching PDF document with Solr, Tika, and Tesseract
  ☆32Updated last year
• alephdata / synonames
  Trying to generate name synonyms from wikidata
  ☆34Updated 5 years ago
• alephdata / datadesktop
  DEPRECATED. Desktop graph visualization application
  ☆51Updated 3 years ago
• DHARPA-Project / kiara
  Data orchestration and management.
  ☆11Updated 2 months ago