tballison / commoncrawl-fetcher-lite

Related projects ⓘ

Alternatives and complementary repositories for commoncrawl-fetcher-lite

• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆63Updated last week
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆65Updated 10 months ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 7 years ago
• mitre / rhapsode
  Advanced desktop search/corpus exploration prototype
  ☆21Updated 3 years ago
• maxdotio / neural-solr
  Neural Solr = Solr 9 + Mighty Inference + Node
  ☆16Updated 2 years ago
• indexdata / metaproxy
  Z39.50/SRU router
  ☆15Updated 2 months ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆24Updated 7 years ago
• thammegowda / tika-ner-corenlp
  Stanford CoreNLP NER addon for Apache Tika's NamerEntityParser
  ☆13Updated 2 years ago
• soprasteria / cybersecurity-dfm
  Data Feed Manager (news watch orchestrator to predict topic with deepdetect and store cleaned text in elasticsearch)
  ☆40Updated 2 years ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆155Updated this week
• opensemanticsearch / solr-relevance-ranking-analysis
  Solr Relevance Ranking Analysis and Visualization Tool
  ☆17Updated 5 years ago
• o19s / pdf-discovery-demo
  Demonstration of searching PDF document with Solr, Tika, and Tesseract
  ☆30Updated last month
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆69Updated 6 months ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆50Updated 4 years ago
• Marcnuth / deduplication
  Remove duplicate documents/videos/images via popular algorithms such as SimHash, SpotSig, Shingling, etc.
  ☆16Updated last year
• iipc / webarchive-commons
  Common web archive utility code.
  ☆50Updated last month
• flaxsearch / harahachibu
  An HTTP proxy for Elasticsearch, Solr (etc.) to prevent a 100% full disk situation.
  ☆11Updated 6 years ago
• w3c / credweb
  Credible Web CG Admin/General
  ☆24Updated 2 years ago
• Sotera / newman
  Quickly analyze and explore email with advanced analytics and visualization.
  ☆55Updated 3 years ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆42Updated 6 years ago
• commoncrawl / cc-webgraph
  Tools to construct and process webgraphs from Common Crawl data
  ☆80Updated this week
• tomcarver / pdf-tools
  A collection of PDF parsing/manipulation tools in Python
  ☆18Updated 13 years ago
• natliblux / warc-safe
  A tool for detecting viruses and NSFW material in WARC files
  ☆11Updated 3 months ago
• bigger312 / Sanctions-Lists
  List of Sanctions and Most wanted
  ☆26Updated 7 years ago
• francelabs / datafari
  Open Source, Distributed, Big Data Enterprise Search Engine
  ☆69Updated this week
• firstlookmedia / dangerzone-converter
  dangerzone has moved to https://github.com/freedomofpress/dangerzone
  ☆40Updated 3 years ago
• tballison / quaerite
  Search relevance evaluation toolkit
  ☆30Updated 2 years ago
• opensanctions / storyweb
  Extract networks of entities from journalistic reporting
  ☆47Updated last year
• strozfriedberg / liblightgrep
  Fast multipattern regular expression searching for digital forensics
  ☆17Updated 5 years ago
• inferlink / landmark-extractor
  ☆12Updated 5 years ago