ikreymer / cc-index-server

Related projects ⓘ

Alternatives and complementary repositories for cc-index-server

• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆42Updated 6 years ago
• xiaoganghan / wikientities
  Linking Entities in CommonCrawl Dataset onto Wikipedia Concepts
  ☆59Updated 12 years ago
• helgeho / Web2Warc
  An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
  ☆24Updated 7 years ago
• trec-kba / streamcorpus-pipeline
  framework for making streamcorpus data
  ☆11Updated 7 years ago
• treygrainger / semantic-knowledge-graph
  ☆19Updated 6 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆159Updated last month
• fujimotos / TinyFastSS
  An index data structure for approximate string search.
  ☆23Updated 5 years ago
• bkj / wit
  Algorithms for "schema matching"
  ☆25Updated 8 years ago
• idio / json-wikipedia
  Json Wikipedia, contains code to convert the Wikipedia xml dump into a json dump. Questions? https://gitter.im/idio-opensource/Lobby
  ☆17Updated 2 years ago
• rossf7 / wikireverse
  Hadoop jobs for WikiReverse project. Parses Common Crawl data for links to Wikipedia articles.
  ☆38Updated 6 years ago
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆42Updated 5 years ago
• eXascaleInfolab / TRank
  Ranking Entity Types using the Web of Data
  ☆30Updated 8 years ago
• dbpedia / distributed-extraction-framework
  DBpedia Distributed Extraction Framework: Extract structured data from Wikipedia in a parallel, distributed manner
  ☆41Updated 2 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆50Updated 4 years ago
• centic9 / CommonCrawlDocumentDownload
  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-test…
  ☆63Updated this week
• uhh-lt / targer
  A web application tagging and retrieval of arguments in text
  ☆30Updated last year
• USCDataScience / autoextractor
  A toolkit for clustering web pages based on various similarity measures.
  ☆32Updated 3 years ago
• knowitall / implie
  Implicit relation extractor using a natural language model.
  ☆25Updated 6 years ago
• inventaire / entities-search-engine
  Scripts and microservice to feed an ElasticSearch with Wikidata and Inventaire entities, and keep those up-to-date
  ☆41Updated 3 years ago
• summa-platform / summa-oss
  Meta-repository for the open-source version of the SUMMA Platform
  ☆16Updated 7 months ago
• maxdotio / neural-solr
  Neural Solr = Solr 9 + Mighty Inference + Node
  ☆16Updated 2 years ago
• Sotera / newman
  Quickly analyze and explore email with advanced analytics and visualization.
  ☆55Updated 3 years ago
• vespa-cloud / cord-19-search
  Vespa application making an index of the CORD-19 dataset.
  ☆39Updated this week
• xtannier / WebAnnotator
  WebAnnotator is a tool for annotating Web pages. WebAnnotator is implemented as a Firefox extension (https://addons.mozilla.org/en-US/fi…
  ☆48Updated 2 years ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆65Updated 10 months ago
• zelandiya / maui-standalone
  ☆21Updated 6 years ago
• trec-kba / streamcorpus
  common data interchange format for document processing pipelines that apply natural language processing tools to large streams of text
  ☆34Updated 8 years ago
• iptc / extra
  Homebase of the IPTC EXTRA project about rule-based text categorization
  ☆13Updated 7 years ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆181Updated 6 years ago
• dbpedia / topicmodel-extractor
  A repository for the "Combining DBpedia and Topic Modeling" GSoC 2016 idea
  ☆13Updated 8 years ago