Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆68Updated 3 weeks ago
• nichtich / jq-wikidata
  jq module to process Wikidata JSON format
  ☆11Updated 6 years ago
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆37Updated 2 years ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 2 years ago
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆31Updated last week
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆68Updated 4 months ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• casework / CASE
  Cyber-investigation Analysis Standard Expression (CASE) Ontology
  ☆70Updated last month
• helgeho / ArchiveSpark
  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed…
  ☆150Updated last month
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆19Updated 11 months ago
• carlospolop / github_archive_scraper
  ☆17Updated last year
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆66Updated 2 weeks ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• ail-project / ail-feeder-twitter
  External twitter feeder for AIL framework
  ☆16Updated 2 years ago
• alephdata / datadesktop
  DEPRECATED. Desktop graph visualization application
  ☆51Updated 2 years ago
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆15Updated 11 months ago
• oduwsdl / MemGator
  A Memento Aggregator CLI and Server in Go
  ☆65Updated 4 months ago
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 2 years ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆55Updated this week
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆92Updated 2 weeks ago
• o19s / pdf-discovery-demo
  Demonstration of searching PDF document with Solr, Tika, and Tesseract
  ☆31Updated 8 months ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆178Updated 6 months ago
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• csirt-tooling-org / csirt-tooling-best-practices
  CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools
  ☆16Updated 3 years ago
• webrecorder / public-web-archives
  A listing of world wide web archives, for humans and machines using Web Archive Manifest (WAM) yaml format
  ☆53Updated 2 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆117Updated last week
• bellingcat / whisperbox-transcribe
  Easy to deploy API for transcribing and translating audio / video using OpenAI's whisper model.
  ☆69Updated last year
• brimstone / twitter-lists
  Auto updating archive of my Twitter lists.
  ☆15Updated 7 months ago
• OpenDataology / OpenDataology
  Best practices for AI dataset metadata and license compliance
  ☆35Updated 2 years ago