Alternatives and similar repositories for CmonCrawl:

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆135Updated 2 months ago
• spider-rs / spider-py
  Spider ported to Python
  ☆68Updated last month
• ilyalasy / DOM-LM
  Unofficial Pytorch implementation of Dom-LM paper.
  ☆33Updated 2 years ago
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆116Updated this week
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆100Updated 3 months ago
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated 6 months ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆113Updated last week
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆26Updated last year
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated 7 months ago
• google / unisim
  UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
  ☆132Updated 3 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆175Updated this week
• zafercavdar / fasttext-langdetect
  80x faster and 95% accurate language identification with Fasttext
  ☆150Updated last year
• bethelmelesse / UnifiedCrawl
  ☆11Updated 3 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆124Updated 2 months ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆49Updated 2 weeks ago
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆49Updated last year
• ScrapeGraphAI / Scrapontologies
  Python library for Entities, relationships and schemas extraction from documents
  ☆37Updated 3 months ago
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆34Updated 6 months ago
• commoncrawl / cc-webgraph
  Tools to construct and process webgraphs from Common Crawl data
  ☆87Updated last week
• leogao2 / commoncrawl_downloader
  ☆33Updated last year
• knowsuchagency / struct-gpt
  get structured output from LLM's
  ☆32Updated last year
• microsoft / dataflow2text
  Code for "The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction an…
  ☆10Updated 10 months ago
• raphaelsty / neural-tree
  Tree-based indexes for neural-search
  ☆29Updated last year
• michaelharms / comcrawl
  A python utility for downloading Common Crawl data
  ☆234Updated last year
• Muhtasham / summarization-eval
  📝 Reference-Free automatic summarization evaluation with potential hallucination detection
  ☆100Updated last year
• voyage-ai / voyageai-python
  Voyage AI Official Python Library
  ☆53Updated 3 months ago
• emrgnt-cmplxty / SmolTrainer
  ☆20Updated last year
• freedmand / interpogate
  A visual tool to interpret and understand PyTorch machine learning models
  ☆16Updated last year
• HawkClaws / main_content_extractor
  A library to extract the main content from html. Developed for information on LLM and for feeding data into LangChain and LlamaIndex.
  ☆35Updated 10 months ago