commoncrawl / cc-pyspark
Process Common Crawl data with Python and Spark
☆422Updated last month
Alternatives and similar repositories for cc-pyspark:
Users that are interested in cc-pyspark are comparing it to the libraries listed below
- Index Common Crawl archives in tabular format☆113Updated 2 weeks ago
- A python utility for downloading Common Crawl data☆236Updated last year
- A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/☆189Updated 6 years ago
- A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine☆169Updated 2 months ago
- Demonstration of using Python to process the Common Crawl dataset with the mrjob framework☆166Updated 2 years ago
- News crawling with StormCrawler - stores content as WARC☆339Updated last month
- Various Jupyter notebooks about Common Crawl data☆51Updated last month
- Heuristic based boilerplate removal tool☆764Updated last month
- Article extraction benchmark: dataset and evaluation scripts☆307Updated 11 months ago
- Python port of Boilerpipe library☆86Updated 7 months ago
- 🌸 fastText + Bloom embeddings for compact, full-coverage vectors with spaCy☆309Updated last year
- Tools to download and cleanup Common Crawl data☆993Updated last year
- NER toolkit for HTML data☆259Updated 10 months ago
- Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters☆135Updated 2 months ago
- SpikeX - SpaCy Pipes for Knowledge Extraction☆398Updated 3 years ago
- Source code for the paper "Web2Text: Deep Structured Boilerplate Removal", full paper @ ECIR'18☆168Updated 3 years ago
- Fast and robust date extraction from web pages, with Python or on the command-line☆123Updated 2 months ago
- Company Name Processor written in Python☆335Updated 10 months ago
- Python tools for interacting with Wikidata☆152Updated last year
- Self-Supervision for Named Entity Disambiguation at the Tail☆215Updated 2 years ago
- Simhash and near-duplicate detection☆413Updated last year
- LexRank algorithm for text summarization☆231Updated 11 months ago
- Extract text from HTML☆134Updated 4 years ago
- Ultimate Website Sitemap Parser☆197Updated last week
- Fuzzy matching and more functionality for spaCy.☆256Updated 8 months ago
- KnowledgeNet: A Benchmark Dataset for Knowledge Base Population☆268Updated 3 years ago
- Database Reasoning Over Text project for ACL paper☆354Updated 2 years ago
- A python based HTML to text conversion library, command line client and Web service.☆297Updated this week
- 💙 Emoji handling and meta data for spaCy with custom extension attributes☆181Updated last year
- DBpedia Spotlight is a tool for automatically annotating mentions of DBpedia resources in text. Improving Efficiency and Accuracy in Mult…☆180Updated 2 years ago