Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆53Updated 2 weeks ago
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 2 years ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 2 years ago
• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆70Updated last week
• Lookyloo / PlaywrightCapture
  Capture a URL with Playwright
  ☆30Updated this week
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆34Updated last month
• Volifter / Belligcat-Hackathon
  ☆21Updated 11 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆188Updated 2 weeks ago
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• niczem / trawler
  scraper for facebook, gab, google and tiktok
  ☆21Updated 2 months ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆66Updated last week
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆36Updated 2 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆183Updated 8 months ago
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆18Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆93Updated last week
• soprasteria / cybersecurity-dfm
  Data Feed Manager (news watch orchestrator to predict topic with deepdetect and store cleaned text in elasticsearch)
  ☆40Updated 2 years ago
• karust / gogetcrawl
  Extract web archive data using Wayback Machine and Common Crawl
  ☆159Updated 9 months ago
• csirt-tooling-org / csirt-tooling-best-practices
  CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools
  ☆16Updated 3 years ago
• alephdata / alephclient
  API client for Aleph, supports bulk entity and document upload.
  ☆28Updated 10 months ago
• bellingcat / whisperbox-transcribe
  Easy to deploy API for transcribing and translating audio / video using OpenAI's whisper model.
  ☆71Updated last year
• opensanctions / storyweb
  Extract networks of entities from journalistic reporting
  ☆48Updated 2 years ago
• nesg-ugr / c4darknet
  ☆10Updated last year
• gephi / viz-engine
  Work in progress: a new visualization engine
  ☆34Updated 2 weeks ago
• pdf-association / pdf-corpora
  An index of PDF-centric corpora
  ☆136Updated last month
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆131Updated last month
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆78Updated last year
• vu-ls / advise
  The Advanced Vulnerability Information Sharing Environment -- A Platform for CVD and PSIRT Management
  ☆10Updated last year
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• ail-project / Similarius
  Similarius is a Python library to compare web page and evaluate the level of similarity.
  ☆21Updated last week