Alternatives and similar repositories for cc-index-table:

Users that are interested in cc-index-table are comparing it to the libraries listed below

• commoncrawl / cc-pyspark
  Process Common Crawl data with Python and Spark
  ☆428Updated 2 months ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆189Updated 6 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆169Updated 3 months ago
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆90Updated 2 weeks ago
• michaelharms / comcrawl
  A python utility for downloading Common Crawl data
  ☆237Updated last year
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆125Updated 3 months ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆410Updated 4 months ago
• commoncrawl / cc-mrjob
  Demonstration of using Python to process the Common Crawl dataset with the mrjob framework
  ☆166Updated 3 years ago
• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆194Updated 7 years ago
• commoncrawl / nutch
  Common Crawl fork of Apache Nutch
  ☆33Updated 3 weeks ago
• ikreymer / webarchive-indexing
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.
  ☆46Updated 7 years ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆312Updated last year
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆52Updated 3 weeks ago
• opensanctions / fingerprints
  Make it easier to compare and cross-reference the names of companies and people by applying strong normalisation.
  ☆150Updated 2 months ago
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆137Updated 3 months ago
• miso-belica / jusText
  Heuristic based boilerplate removal tool
  ☆766Updated last month
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 2 years ago
• commoncrawl / news-crawl
  News crawling with StormCrawler - stores content as WARC
  ☆344Updated 2 months ago
• ikreymer / cc-index-server
  Deployment of pywb as a CommonCrawl Index Server
  ☆21Updated 7 years ago
• jmriebold / BoilerPy3
  Python port of Boilerpipe library
  ☆86Updated 8 months ago
• commoncrawl / cc-index-server
  Common Crawl Index Server
  ☆68Updated last month
• mediacloud / date_guesser
  A library to extract a publication date from a web page, along with a measure of the accuracy.
  ☆41Updated 5 years ago
• kermitt2 / entity-fishing
  A machine learning tool for fishing entities
  ☆263Updated 3 weeks ago
• hynky1999 / CmonCrawl
  Common crawl extractor
  ☆75Updated 11 months ago
• dalab / web2text
  Source code for the paper "Web2Text: Deep Structured Boilerplate Removal", full paper @ ECIR'18
  ☆169Updated 3 years ago
• UB-Mannheim / spacyopentapioca
  A spaCy wrapper of OpenTapioca for named entity linking on Wikidata
  ☆94Updated 2 years ago
• divkakwani / awesome-newspapers
  A Directory of Online Newspaper Sources for 70+ Languages
  ☆32Updated 4 years ago
• opensanctions / yente
  API for OpenSanctions with support for entity search and bulk matching of data collections. Supports Reconciliation API spec.
  ☆81Updated this week
• weblyzard / inscriptis
  A python based HTML to text conversion library, command line client and Web service.
  ☆302Updated last month
• ukwa / webarchive-discovery
  WARC and ARC indexing and discovery tools.
  ☆123Updated last month