Alternatives and similar repositories for CmonCrawl

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• google / unisim
  UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
  ☆137Updated 3 months ago
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆28Updated last year
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆184Updated last week
• TuanaCelik / unstructuredio-haystack
  💙 Unstructured Data Connectors for Haystack 2.0
  ☆17Updated last year
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆92Updated last week
• dwarvesf / llm-hosting
  This repository is designed for deploying and managing server processes that handle embeddings using the Infinity Embedding model or Larg…
  ☆23Updated 4 months ago
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆142Updated 6 months ago
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆127Updated last week
• browser-use / eval
  ☆34Updated 5 months ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆84Updated last month
• databyjp / distyll
  A personal knowledge base that I can dump information to and help me learn
  ☆24Updated last month
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆50Updated 9 months ago
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆37Updated 2 months ago
• S1M0N38 / dspy-arxiv
  Explore the use of DSPy for extracting features from PDFs 🔎
  ☆43Updated last year
• networkdynamics / seldonite
  A News Article Collection Library
  ☆22Updated 2 years ago
• nomic-ai / semantic-search-app-template
  Tutorial and template for a semantic search app powered by the Atlas Embedding Database, Langchain, OpenAI and FastAPI
  ☆115Updated last year
• OpenPipe / pii-redaction
  Detect and redact PII locally with SOTA performance
  ☆58Updated 3 months ago
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆54Updated last year
• neoxelox / dspy-inspector
  DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.
  ☆36Updated last year
• neuml / txtmarker
  🖍️ Highlight text in documents
  ☆109Updated 2 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆133Updated 6 months ago
• masci / banks
  LLM prompt language based on Jinja. Banks provides tools and functions to build prompts text and chat messages from generic blueprints. I…
  ☆105Updated 2 weeks ago
• DeployQL / LintDB
  Vector Database with support for late interaction and token level embeddings.
  ☆55Updated 3 weeks ago
• hwchase17 / chain-of-verification
  ☆32Updated last year
• weaviate / structured-rag
  Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models
  ☆108Updated 3 months ago
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆54Updated 3 months ago
• Chirayu-Tripathi / nl2query
  A framework for converting natural language text inputs to corresponding Pandas, MongoDB, Kusto and Neo4j (Cypher) queries.
  ☆83Updated last year
• laura-ham / HM-Fashion-image-neural-search
  H&M Fashion Image similarity search with Weaviate and DocArray
  ☆43Updated last year
• voyage-ai / voyageai-python
  Voyage AI Official Python Library
  ☆61Updated 2 weeks ago
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆111Updated 3 months ago