Alternatives and similar repositories for cc-mrjob:

Users that are interested in cc-mrjob are comparing it to the libraries listed below

• trivio / common_crawl_index
  Index URLs in Common Crawl
  ☆194Updated 7 years ago
• ikreymer / cdx-index-client
  A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
  ☆189Updated 6 years ago
• scrapinghub / webstruct
  NER toolkit for HTML data
  ☆259Updated 11 months ago
• scrapinghub / aile
  Automatic Item List Extraction
  ☆87Updated 8 years ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆117Updated last month
• TeamHG-Memex / Formasaurus
  Formasaurus tells you the type of an HTML form and its fields using machine learning
  ☆118Updated 10 months ago
• turian / topia.termextract
  Updates to Zope's keyphrase extractor (forked from 1.1.0)
  ☆67Updated 7 years ago
• commoncrawl / cc-pyspark
  Process Common Crawl data with Python and Spark
  ☆428Updated 2 months ago
• edsu / microdata
  python library for extracting html microdata
  ☆166Updated last year
• explosion / displacy-ent
  displaCy-ent.js: An open-source named entity visualiser for the modern web
  ☆199Updated 7 years ago
• TeamHG-Memex / deep-deep
  Adaptive crawler which uses Reinforcement Learning methods
  ☆169Updated 6 years ago
• GregBowyer / cld2-cffi
  Python bindings to the Compact Language Detector
  ☆33Updated 4 years ago
• rossf7 / elasticrawl
  Launch AWS Elastic MapReduce jobs that process Common Crawl data.
  ☆49Updated 8 years ago
• tristantao / py-ms-cognitive
  Thin wrapper for the Microsoft Cognitive Services
  ☆60Updated 7 years ago
• nickdavidhaynes / spacy-cld
  Language detection extension for spaCy 2.0+
  ☆112Updated 6 years ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆169Updated 3 months ago
• explosion / spacy-dev-resources
  💫 Scripts, tools and resources for developing spaCy
  ☆126Updated 6 years ago
• scrapinghub / page_clustering
  A simple algorithm for clustering web pages, suitable for crawlers
  ☆34Updated 8 years ago
• dat / pyner
  Python interface to the Stanford Named Entity Recognizer
  ☆292Updated 3 years ago
• rossf7 / wikireverse
  Hadoop jobs for WikiReverse project. Parses Common Crawl data for links to Wikipedia articles.
  ☆38Updated 6 years ago
• scrapinghub / mdr
  A python library detect and extract listing data from HTML page.
  ☆108Updated 7 years ago
• misja / python-boilerpipe
  Python interface to Boilerpipe, Boilerplate Removal and Fulltext Extraction from HTML pages
  ☆544Updated 3 years ago
• Lab41 / pythia
  Supervised learning for novelty detection in text
  ☆78Updated 8 years ago
• Webhose / article-date-extractor
  Automatically extracts and normalizes an online article or blog post publication date
  ☆117Updated last year
• internetarchive / warc
  Python library for reading and writing warc files
  ☆239Updated 3 years ago
• dkpro / dkpro-c4corpus
  DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate…
  ☆52Updated 4 years ago
• piskvorky / gensim-simserver
  [NO LONGER MAINTAINED AS OPEN SOURCE - USE SCALETEXT.COM INSTEAD]
  ☆108Updated 11 years ago
• webrecorder / warcio
  Streaming WARC/ARC library for fast web archive IO
  ☆410Updated 4 months ago
• pavlobaron / wpcorpus
  wpcorpus - NLP corpus based on Wikipedia's full article dump
  ☆97Updated 9 years ago
• cocrawler / cocrawler
  CoCrawler is a versatile web crawler built using modern tools and concurrency.
  ☆190Updated 2 years ago