Alternatives and similar repositories for CmonCrawl

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆137Updated 4 months ago
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆51Updated last year
• spider-rs / spider-py
  Spider ported to Python
  ☆82Updated 3 months ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆60Updated last month
• argilla-io / awesome-llm-datasets
  👩🤝🤖 A curated list of datasets for large language models (LLMs), RLHF and related resources (continually updated)
  ☆23Updated 2 years ago
• OpenPipe / pii-redaction
  Detect and redact PII locally with SOTA performance
  ☆47Updated last month
• x-tabdeveloping / neofuzz
  Blazing fast fuzzy text search for Python.
  ☆43Updated 3 weeks ago
• weaviate-tutorials / Hurricane
  Writing Blog Posts with Generative Feedback Loops!
  ☆47Updated last year
• dstackai / gpuhunt
  GPU prices aggregator for cloud providers
  ☆37Updated 2 weeks ago
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆53Updated last month
• explodinggradients / Funtuner
  Supervised instruction finetuning for LLM with HF trainer and Deepspeed
  ☆35Updated last year
• davanstrien / haiku-dpo
  Using open source LLMs to build synthetic datasets for direct preference optimization
  ☆61Updated last year
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆126Updated 4 months ago
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆35Updated 2 weeks ago
• louisbrulenaudet / ragoon
  High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡
  ☆66Updated 6 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆178Updated last week
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆62Updated this week
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆107Updated last month
• emrgnt-cmplxty / SmolTrainer
  ☆20Updated last year
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆50Updated 7 months ago
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆90Updated last week
• mithril-security / blind_llama_client
  Zero-trust AI APIs for easy and private consumption of open-source LLMs
  ☆40Updated 9 months ago
• lightonai / ducksearch
  Efficient BM25 with DuckDB 🦆
  ☆48Updated 4 months ago
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• kenhktsui / anyclassifier
  One Line To Build Zero-Data Classifiers in Minutes
  ☆54Updated 7 months ago
• S1M0N38 / dspy-arxiv
  Explore the use of DSPy for extracting features from PDFs 🔎
  ☆39Updated last year
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆123Updated last week
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated 9 months ago
• knowsuchagency / struct-gpt
  get structured output from LLM's
  ☆33Updated 2 years ago
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆28Updated last year