Alternatives and similar repositories for commoncrawl-fetcher-lite:

Users that are interested in commoncrawl-fetcher-lite are comparing it to the libraries listed below

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆64Updated 2 months ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 7 years ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆25Updated 7 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆44Updated 7 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 3 years ago
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆11Updated 6 months ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆66Updated 2 weeks ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆53Updated 2 months ago
• nichtich / jq-wikidata
  jq module to process Wikidata JSON format
  ☆11Updated 5 years ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆171Updated 2 weeks ago
• thammegowda / tika-ner-corenlp
  Stanford CoreNLP NER addon for Apache Tika's NamerEntityParser
  ☆13Updated 2 years ago
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆74Updated 9 months ago
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆121Updated 6 months ago
• indexdata / metaproxy
  Z39.50/SRU router
  ☆15Updated this week
• opensemanticsearch / solr-relevance-ranking-analysis
  Solr Relevance Ranking Analysis and Visualization Tool
  ☆17Updated 5 years ago
• netarchivesuite / solrwayback
  A search interface and wayback machine for the UKWA Solr based warc-indexer framework.
  ☆108Updated last month
• commoncrawl / cc-webgraph
  Tools to construct and process webgraphs from Common Crawl data
  ☆85Updated 3 weeks ago
• exponential-decay / the-format-registry
  A mirror of the PRONOM file format registry in Linked Open Data format. The Format Registry is a linked (open) data file format repositor…
  ☆10Updated last year
• digital-scrappy / network-analysis-hackathon
  Template repository and README for submissions to Bellingcat's Global Hackathon
  ☆16Updated 2 years ago
• hybridtheory / floc-simhash
  A fast python implementation of the SimHash algorithm.
  ☆27Updated 3 years ago
• investigativedata / investigraph
  etl pipeline, graphical explorer and general toolbox for investigations with follow the money data
  ☆15Updated last year
• rholder / nilsimsa
  This module contains an implementation of the Nilsimsa locality-sensitive hashing algorithm in Java.
  ☆18Updated 5 years ago
• alephdata / ingest-file
  Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.
  ☆58Updated this week
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆166Updated last month
• webrecorder / cdxj-indexer
  CDXJ Indexing of WARC/ARCs
  ☆25Updated 2 months ago
• cvcio / rtaa-72
  RTAA-72, is CVCIO's real-time intelligence dashboard for Twitter
  ☆21Updated 2 years ago
• httpreserve / httpreserve
  Digital Preservation of HTTP in documentary heritage.
  ☆22Updated last year
• chryzsh / GPTCommentDetector
  A UserScript to detect GPT generated comments on Hackernews.
  ☆13Updated 2 years ago
• Marcnuth / deduplication
  Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
  ☆18Updated last year