Alternatives and similar repositories for CmonCrawl:

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆137Updated 3 months ago
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆51Updated last year
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆121Updated this week
• louisbrulenaudet / ragoon
  High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡
  ☆66Updated 5 months ago
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆27Updated last year
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆177Updated last week
• ilyalasy / DOM-LM
  Unofficial Pytorch implementation of Dom-LM paper.
  ☆33Updated 2 years ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆56Updated 2 weeks ago
• cisnlp / GlotLID
  💬 Language Identification with Support for More Than 2000 Labels -- EMNLP 2023
  ☆127Updated 4 months ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆312Updated last year
• MoritzLaurer / zeroshot-classifier
  Notebooks for training universal 0-shot classifiers on many different tasks
  ☆124Updated 3 months ago
• sdtblck / youtube_subtitle_dataset
  YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training
  ☆43Updated 4 years ago
• neuml / staticvectors
  🔢 Work with static vector models
  ☆28Updated this week
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆103Updated 3 weeks ago
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆62Updated 10 months ago
• kddubey / cappr
  Completion After Prompt Probability. Make your LLM make a choice
  ☆76Updated 5 months ago
• serpapi / serpapi-python
  a Python client library for SerpApi.
  ☆84Updated 9 months ago
• Knowledgator / unlimited_classifier
  Universal text classifier for generative models
  ☆24Updated 9 months ago
• argilla-io / awesome-llm-datasets
  👩🤝🤖 A curated list of datasets for large language models (LLMs), RLHF and related resources (continually updated)
  ☆23Updated last year
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆52Updated 3 weeks ago
• weaviate / t2v-transformers-models
  This is the repo for the container that holds the models for the text2vec-transformers module
  ☆51Updated 3 weeks ago
• flairNLP / fabricator
  [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.
  ☆108Updated 11 months ago
• raphaelsty / neural-tree
  Tree-based indexes for neural-search
  ☆31Updated last year
• ScrapeGraphAI / Scrapontologies
  Python library for Entities, relationships and schemas extraction from documents
  ☆38Updated 4 months ago
• alexi / juno
  ☆28Updated last year
• Webhose / free-news-datasets
  Weekly free datasets from global news sites
  ☆22Updated this week
• emrgnt-cmplxty / SmolTrainer
  ☆20Updated last year
• khaimt / qa_expert
  This repo is for handling Question Answering, especially for Multi-hop Question Answering
  ☆67Updated last year
• jina-ai / jerboa
  LLM finetuning
  ☆42Updated last year
• addy999 / omniparser-api
  Self-hosted version of Microsoft's OmniParser Image-to-text model
  ☆64Updated 4 months ago