Alternatives and similar repositories for CmonCrawl

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆188Updated last week
• x-tabdeveloping / neofuzz
  Blazing fast fuzzy text search for Python.
  ☆46Updated 4 months ago
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆143Updated 8 months ago
• spider-rs / spider-py
  Spider ported to Python
  ☆89Updated 7 months ago
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆53Updated last week
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆137Updated this week
• networkdynamics / seldonite
  A News Article Collection Library
  ☆22Updated 2 years ago
• google / unisim
  UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
  ☆139Updated 4 months ago
• weaviate / t2v-transformers-models
  This is the repo for the container that holds the models for the text2vec-transformers module
  ☆54Updated 3 weeks ago
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated 11 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆138Updated 3 weeks ago
• nlpcloud / nlpcloud-python
  NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, …
  ☆84Updated 9 months ago
• voyage-ai / voyageai-python
  Voyage AI Official Python Library
  ☆71Updated last month
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆28Updated last year
• masci / banks
  LLM prompt language based on Jinja. Banks provides tools and functions to build prompts text and chat messages from generic blueprints. I…
  ☆114Updated last month
• parkervg / blendsql
  Query language for blending SQL and LLMs across structured + unstructured data, with type constraints.
  ☆108Updated last week
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆321Updated last year
• advanced-stack / py-llm-core
  A pythonic library providing light-weighted interface with LLMs
  ☆128Updated 3 months ago
• lancedb / aifunctools
  Utility for OpenAI GPT Functions
  ☆14Updated 2 years ago
• TuanaCelik / unstructuredio-haystack
  💙 Unstructured Data Connectors for Haystack 2.0
  ☆17Updated last year
• neuml / txtmarker
  🖍️ Highlight text in documents
  ☆108Updated 4 months ago
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆54Updated 4 months ago
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated last week
• msoedov / vector_lake
  S3 vector database for LLM Agents and RAG.
  ☆48Updated 2 years ago
• vespa-engine / vespa-search
  ☆12Updated last week
• currentslab / extractnet
  A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one pac…
  ☆292Updated 3 months ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆84Updated 2 months ago
• Chirayu-Tripathi / nl2query
  A framework for converting natural language text inputs to corresponding Pandas, MongoDB, Kusto and Neo4j (Cypher) queries.
  ☆88Updated last year
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆25Updated last year
• Sripaad / ai4privacy
  ☆19Updated last year