Alternatives and similar repositories for CmonCrawl

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• ilyalasy / DOM-LM
  Unofficial Pytorch implementation of Dom-LM paper.
  ☆33Updated 2 years ago
• karust / gogetcrawl
  Extract web archive data using Wayback Machine and Common Crawl
  ☆157Updated 7 months ago
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆111Updated 2 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆183Updated 3 weeks ago
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆127Updated this week
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆79Updated 3 weeks ago
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆122Updated last month
• google / unisim
  UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
  ☆137Updated 2 months ago
• kddubey / cappr
  Completion After Prompt Probability. Make your LLM make a choice
  ☆79Updated 7 months ago
• domanchi / gibberish-detector
  Train a model, and detect gibberish strings with it.
  ☆62Updated 3 years ago
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆141Updated 5 months ago
• pinecone-io / pinecone-text
  Pinecone text client library
  ☆62Updated 3 months ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆177Updated 5 months ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆317Updated last year
• nlpcloud / nlpcloud-python
  NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, …
  ☆82Updated 6 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆130Updated 5 months ago
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆54Updated 2 months ago
• Snowflake-Labs / arctic-embed
  ☆72Updated 6 months ago
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆53Updated last year
• x-tabdeveloping / neofuzz
  Blazing fast fuzzy text search for Python.
  ☆44Updated 2 months ago
• knowsuchagency / struct-gpt
  get structured output from LLM's
  ☆32Updated 2 years ago
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆37Updated last month
• bethelmelesse / UnifiedCrawl
  ☆13Updated 7 months ago
• michaelharms / comcrawl
  A python utility for downloading Common Crawl data
  ☆240Updated 2 years ago
• networkdynamics / seldonite
  A News Article Collection Library
  ☆23Updated 2 years ago
• currentslab / extractnet
  A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one pac…
  ☆284Updated last month
• EdyVision / pii-codex
  A research python package for detecting, categorizing, and assessing the severity of personal identifiable information (PII)
  ☆87Updated last year
• neuml / staticvectors
  🔢 Work with static vector models
  ☆28Updated 2 months ago
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆28Updated last year
• DeployQL / LintDB
  Vector Database with support for late interaction and token level embeddings.
  ☆55Updated this week