Alternatives and similar repositories for cc-index-server

Users that are interested in cc-index-server are comparing it to the libraries listed below

• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆189Updated 3 weeks ago
• CI-Research / KeywordAnalysis
  Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
  ☆57Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆102Updated this week
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆205Updated 7 years ago
• VIDA-NYU / domain_discovery_tool
  This repository contains the Domain Discovery Tool (DDT) project. DDT is an interactive system that helps users explore and better unders…
  ☆47Updated 4 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆124Updated last week
• anvaka / yasiv-youtube
  Graph of related videos from YouTube
  ☆111Updated 3 years ago
• iproduct-database / vpm-filter-spark
  Virtual patent marking crawler at iproduct.epfl.ch
  ☆15Updated 8 years ago
• commoncrawl / cc-pyspark
  Process Common Crawl data with Python and Spark
  ☆449Updated last month
• opensemanticsearch / solr-relevance-ranking-analysis
  Solr Relevance Ranking Analysis and Visualization Tool
  ☆15Updated 6 years ago
• medialab / hyphe-browser
  Browser version of Hyphe (WIP)
  ☆32Updated 6 months ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 8 years ago
• openpreserve / pagelyzer
  Suite of tools for detecting changes in web pages and their rendering
  ☆55Updated last year
• idio / json-wikipedia
  Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump. Questions? https://gitter.im/idio-opensource/Lobby
  ☆17Updated 3 years ago
• darpa-i2o / memex-program-index
  A list of memex-related tools and their repository URLs
  ☆155Updated 7 years ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 8 years ago
• alephdata / alephclient
  API client for Aleph, supports bulk entity and document upload.
  ☆28Updated last year
• tballison / commoncrawl-fetcher-lite
  Simplified version of a common crawl fetcher
  ☆17Updated last week
• carlospolop / github_archive_scraper
  ☆18Updated 2 years ago
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 8 years ago
• VIDA-NYU / ache
  ACHE is a web crawler for domain-specific search.
  ☆474Updated 3 months ago
• iipc / webarchive-commons
  Common web archive utility code.
  ☆57Updated last week
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 5 years ago
• opensemanticsearch / open-semantic-etl
  Python based Open Source ETL tools for file crawling, document processing (text extraction, OCR), content analysis (Entity Extraction & N…
  ☆275Updated 3 years ago
• GateNLP / gate-core
  The GATE Embedded core API and GATE Developer application
  ☆88Updated last year
• terrier-org / terrier-core
  Terrier IR Platform
  ☆269Updated last week
• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆72Updated 2 weeks ago
• commoncrawl / nutch
  Common Crawl fork of Apache Nutch
  ☆40Updated last week
• rflynn / regroup
  Generate a regular expression that describes a set of strings.
  ☆31Updated 3 years ago
• inventaire / entities-search-engine
  Scripts and microservice to feed an ElasticSearch with Wikidata and Inventaire entities, and keep those up-to-date
  ☆41Updated 4 years ago