Alternatives and similar repositories for CmonCrawl

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆192Updated 2 weeks ago
• x-tabdeveloping / neofuzz
  Blazing fast fuzzy text search for Python.
  ☆47Updated 5 months ago
• networkdynamics / seldonite
  A News Article Collection Library
  ☆22Updated 2 years ago
• commoncrawl / cc-webgraph
  Tools to construct and process Common Crawl webgraphs
  ☆96Updated 3 weeks ago
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆146Updated 8 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆140Updated last month
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated last year
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆122Updated last month
• neuml / txtmarker
  🖍️ Highlight text in documents
  ☆109Updated 4 months ago
• commoncrawl / cc-downloader
  A polite and user-friendly downloader for Common Crawl data
  ☆57Updated last month
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆58Updated 5 months ago
• OpenPipe / pii-redaction
  Detect and redact PII locally with SOTA performance
  ☆72Updated 5 months ago
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆141Updated this week
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆322Updated last year
• google / unisim
  UniSim is a package for efficient similarity computation, fuzzy matching, and clustering of data.
  ☆142Updated 5 months ago
• kddubey / cappr
  Completion After Prompt Probability. Make your LLM make a choice
  ☆80Updated 10 months ago
• spider-rs / spider-py
  Spider ported to Python
  ☆91Updated 7 months ago
• qdrant / goods_categorization_demo
  Demo example of consumer goods categorization
  ☆28Updated last year
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆116Updated 5 months ago
• currentslab / extractnet
  A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one pac…
  ☆293Updated 4 months ago
• nlpcloud / nlpcloud-python
  NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, …
  ☆85Updated 9 months ago
• nomic-ai / semantic-search-app-template
  Tutorial and template for a semantic search app powered by the Atlas Embedding Database, Langchain, OpenAI and FastAPI
  ☆114Updated 2 years ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆85Updated last week
• neuml / txtchat
  💭 Build autonomous agents, retrieval augmented generation (RAG) processes and language model powered chat applications
  ☆297Updated 4 months ago
• parkervg / blendsql
  Query language for blending SQL and LLMs across structured + unstructured data, with type constraints.
  ☆109Updated last week
• teticio / lambda-scraper
  Use AWS Lambda functions as a proxy pool to scrape web pages.
  ☆137Updated last year
• EdyVision / pii-codex
  A research python package for detecting, categorizing, and assessing the severity of personal identifiable information (PII)
  ☆89Updated 3 weeks ago
• JoelNiklaus / LegalDatasets
  This repository serves as a collection of scrapers procuring and structuring various legal datasets
  ☆18Updated 2 years ago
• serpapi / lego-ai-parser
  Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
  ☆236Updated last year
• vespa-engine / vespa-search
  ☆12Updated last week