Alternatives and similar repositories for commoncrawl-fetcher-lite

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆68Updated last week
• nichtich / jq-wikidata
  jq module to process Wikidata JSON format
  ☆11Updated 6 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 4 years ago
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆48Updated last month
• CAIDA / commoncrawl-host-ip-mapper
  Crawler that retrieves commoncrawl's crawled hosts and their corresponding IPs
  ☆18Updated 10 months ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 7 years ago
• adulau / napkin-text-analysis
  Napkin is a simple tool to produce statistical analysis of a text
  ☆12Updated last year
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 2 years ago
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆15Updated 10 months ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• cisagov / flare-misp-service
  Automate the regular transfer of AIS data into a MISP Server
  ☆10Updated last year
• exponential-decay / the-format-registry
  A mirror of the PRONOM file format registry in Linked Open Data format. The Format Registry is a linked (open) data file format repositor…
  ☆10Updated 2 years ago
• niczem / trawler
  scraper for facebook, gab, google and tiktok
  ☆21Updated last week
• datacoon / metawarc
  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)
  ☆30Updated 2 weeks ago
• csirt-tooling-org / csirt-tooling-best-practices
  CSIRT Tooling: Best Practices in Developing, Maintaining and Distributing Open Source Tools
  ☆16Updated 3 years ago
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆76Updated last year
• iipc / webarchive-commons
  Common web archive utility code.
  ☆55Updated last month
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 2 years ago
• chryzsh / GPTCommentDetector
  A UserScript to detect GPT generated comments on Hackernews.
  ☆14Updated 2 years ago
• gcve-eu / gcve
  A Python client for the Global CVE Allocation System.
  ☆13Updated this week
• OpenDataology / OpenDataology
  Best practices for AI dataset metadata and license compliance
  ☆35Updated 2 years ago
• thammegowda / tika-ner-corenlp
  Stanford CoreNLP NER addon for Apache Tika's NamerEntityParser
  ☆13Updated 3 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• ail-project / ail-feeder-twitter
  External twitter feeder for AIL framework
  ☆16Updated 2 years ago
• oduwsdl / MemGator
  A Memento Aggregator CLI and Server in Go
  ☆65Updated 3 months ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆65Updated last week
• archivesunleashed / twut
  An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.
  ☆9Updated 6 months ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆68Updated 3 months ago
• bigger312 / Sanctions-Lists
  List of Sanctions and Most wanted
  ☆28Updated 8 years ago
• Supahands / llm-comparison
  This is an opensource project allowing you to compare two LLM's head to head with a given prompt, it has a wide range of supported models…
  ☆23Updated 3 months ago