Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆199Updated this week
• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆71Updated last week
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆17Updated 3 years ago
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆59Updated 3 months ago
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆20Updated 2 months ago
• soprasteria / cybersecurity-dfm
  Data Feed Manager (news watch orchestrator to predict topic with deepdetect and store cleaned text in elasticsearch)
  ☆40Updated 3 years ago
• pdf-association / pdf-corpora
  An index of PDF-centric corpora
  ☆145Updated 4 months ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆36Updated 3 weeks ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆71Updated 8 months ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆66Updated 3 weeks ago
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆101Updated 2 weeks ago
• Lookyloo / PlaywrightCapture
  Capture a URL with Playwright
  ☆30Updated this week
• Volifter / Belligcat-Hackathon
  ☆21Updated last year
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆187Updated 2 weeks ago
• csirt-tooling-org / csirt-tooling-best-practices
  CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools
  ☆16Updated 3 years ago
• bellingcat / whisperbox-transcribe
  Easy to deploy API for transcribing and translating audio / video using OpenAI's whisper model.
  ☆71Updated last year
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆36Updated 3 years ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 3 years ago
• alephdata / alephclient
  API client for Aleph, supports bulk entity and document upload.
  ☆28Updated last year
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆131Updated last week
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆83Updated last year
• deep-insights-ai / covid-19-classifier
  A fastai2 based Covid-19 classifier
  ☆17Updated 5 years ago
• vulnerability-lookup / VulnTrain
  A tool to generate datasets and models based on vulnerabilities descriptions from @Vulnerability-Lookup.
  ☆20Updated this week
• o19s / pdf-discovery-demo
  Demonstration of searching PDF document with Solr, Tika, and Tesseract
  ☆32Updated last year
• indexdata / metaproxy
  Z39.50/SRU router
  ☆17Updated last month
• opensanctions / yente
  API for OpenSanctions with support for entity search and bulk matching of data collections. Supports Reconciliation API spec.
  ☆111Updated this week
• vulnerability-lookup / PyVulnerabilityLookup
  Python client and module for Vulnerability-Lookup.
  ☆14Updated 3 weeks ago
• ail-project / Similarius
  Similarius is a Python library to compare web page and evaluate the level of similarity.
  ☆21Updated last month