Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆193Updated this week
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆56Updated last month
• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆71Updated last month
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆36Updated 3 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 3 years ago
• Volifter / Belligcat-Hackathon
  ☆21Updated last year
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆70Updated 7 months ago
• DistriNet / tranco-list
  Tranco: An improved top websites ranking
  ☆164Updated 5 years ago
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 3 years ago
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆34Updated 2 months ago
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆19Updated last month
• pdf-association / pdf-corpora
  An index of PDF-centric corpora
  ☆140Updated 3 months ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆184Updated this week
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆17Updated last year
• enferex / pdfresurrect
  Analyze and help extract older "hidden" versions of a pdf from the current pdf.
  ☆86Updated 3 years ago
• thinkst / zippy
  Detect AI-generated text [relatively] quickly via compression ratios
  ☆262Updated 7 months ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆66Updated this week
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• alephdata / datadesktop
  DEPRECATED. Desktop graph visualization application
  ☆51Updated 3 years ago
• csirt-tooling-org / csirt-tooling-best-practices
  CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools
  ☆16Updated 3 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆131Updated last week
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• ail-project / Similarius
  Similarius is a Python library to compare web page and evaluate the level of similarity.
  ☆21Updated last week
• simonrdavies / NapierOne
  NapierOne. A Publicly Available Modern Mixed File Data Set. The data set is suitable for a variety of testing scenarios such as Ransomwar…
  ☆21Updated 3 years ago
• sowdust / pdfxplr
  Extract hidden data from pdf files
  ☆57Updated 6 years ago
• freelawproject / x-ray
  A tool to detect whether a PDF has a bad redaction
  ☆152Updated last month
• sahanmar / 451
  ☆36Updated 3 years ago
• niczem / trawler
  scraper for facebook, gab, google and tiktok
  ☆21Updated 3 months ago
• dfxml-working-group / dfxml_python
  Digital Forensics XML packages in Python
  ☆17Updated 6 months ago