Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆66Updated last month
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆16Updated 9 months ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 3 years ago
• MischaU8 / spevktator
  An open source investigation tool to collect and analyse public VK community wall posts
  ☆36Updated 2 years ago
• Volifter / Belligcat-Hackathon
  ☆21Updated 7 months ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 2 years ago
• carlospolop / github_archive_scraper
  ☆17Updated last year
• thammegowda / tika-ner-corenlp
  Stanford CoreNLP NER addon for Apache Tika's NamerEntityParser
  ☆13Updated 3 years ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆68Updated 2 months ago
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• nichtich / jq-wikidata
  jq module to process Wikidata JSON format
  ☆11Updated 5 years ago
• Lookyloo / PlaywrightCapture
  Capture a URL with Playwright
  ☆29Updated last week
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆24Updated 7 years ago
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆14Updated 9 months ago
• soprasteria / cybersecurity-dfm
  Data Feed Manager (news watch orchestrator to predict topic with deepdetect and store cleaned text in elasticsearch)
  ☆40Updated 2 years ago
• niczem / trawler
  scraper for facebook, gab, google and tiktok
  ☆21Updated 10 months ago
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆29Updated 8 months ago
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆43Updated last week
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 2 years ago
• simonrdavies / NapierOne
  NapierOne. A Publicly Available Modern Mixed File Data Set. The data set is suitable for a variety of testing scenarios such as Ransomwar…
  ☆19Updated 3 years ago
• chryzsh / GPTCommentDetector
  A UserScript to detect GPT generated comments on Hackernews.
  ☆14Updated 2 years ago
• cyclelabscloud / cycle-cloud
  Repository to allow collaboration between Cycle Labs Cloud community in support of the community.
  ☆9Updated 3 years ago
• analyticascent / stylext
  An authorship attribution project with particular emphasis on Twitter analysis
  ☆16Updated 3 years ago
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆90Updated last week
• iipc / webarchive-commons
  Common web archive utility code.
  ☆55Updated 2 months ago
• cygenta / top10million
  A repository of the 10 million live most popular websites
  ☆40Updated 2 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆113Updated 3 weeks ago
• Supahands / llm-comparison
  This is an opensource project allowing you to compare two LLM's head to head with a given prompt, it has a wide range of supported models…
  ☆23Updated last month
• datasette / datasette-enrichments-gpt
  Datasette enrichment for analyzing row data using OpenAI's GPT models
  ☆19Updated last year
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆75Updated 11 months ago