Alternatives and similar repositories for CmonCrawl:

Users that are interested in CmonCrawl are comparing it to the libraries listed below

• spider-rs / spider-py
  Spider ported to Python
  ☆68Updated last month
• commoncrawl / cc-index-table
  Index Common Crawl archives in tabular format
  ☆113Updated this week
• adbar / courlan
  Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
  ☆135Updated 2 months ago
• commoncrawl / cc-crawl-statistics
  Statistics of Common Crawl monthly archives mined from URL index files
  ☆175Updated this week
• argilla-io / awesome-llm-datasets
  👩🤝🤖 A curated list of datasets for large language models (LLMs), RLHF and related resources (continually updated)
  ☆23Updated last year
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆50Updated last year
• vespa-engine / pyvespa
  Python API for https://vespa.ai, the open big data serving engine
  ☆116Updated this week
• networkdynamics / seldonite
  A News Article Collection Library
  ☆22Updated last year
• DeployQL / LintDB
  Vector Database with support for late interaction and token level embeddings.
  ☆53Updated 5 months ago
• cocrawler / cdx_toolkit
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
  ☆168Updated 2 months ago
• Lv-291 / wasm-bhtsne
  Parallel wasm Barnes-Hut t-SNE implementation written in Rust.
  ☆17Updated 9 months ago
• kddubey / cappr
  Completion After Prompt Probability. Make your LLM make a choice
  ☆74Updated 4 months ago
• commoncrawl / cc-notebooks
  Various Jupyter notebooks about Common Crawl data
  ☆51Updated 3 weeks ago
• S1M0N38 / dspy-arxiv
  Explore the use of DSPy for extracting features from PDFs 🔎
  ☆38Updated last year
• chatnoir-eu / chatnoir-resiliparse
  A robust web archive analytics toolkit
  ☆99Updated 3 months ago
• adbar / htmldate
  Fast and robust date extraction from web pages, with Python or on the command-line
  ☆124Updated 2 months ago
• ilyalasy / DOM-LM
  Unofficial Pytorch implementation of Dom-LM paper.
  ☆33Updated 2 years ago
• johnbumgarner / newshound
  This Python package can be used to systematically extract multiple data elements (e.g., title, keywords, text) from news sources around t…
  ☆33Updated last year
• weaviate / t2v-transformers-models
  This is the repo for the container that holds the models for the text2vec-transformers module
  ☆49Updated last month
• wizenheimer / cyyrus
  Transform Unstructured Data into Synthetic Datasets
  ☆26Updated 6 months ago
• scrapinghub / article-extraction-benchmark
  Article extraction benchmark: dataset and evaluation scripts
  ☆305Updated 10 months ago
• masci / banks
  LLM prompt language based on Jinja. Banks provides tools and functions to build prompts text and chat messages from generic blueprints. I…
  ☆81Updated last week
• khaimt / qa_expert
  This repo is for handling Question Answering, especially for Multi-hop Question Answering
  ☆67Updated last year
• TIGER-AI-Lab / StructLM
  Code and data for "StructLM: Towards Building Generalist Models for Structured Knowledge Grounding" (COLM 2024)
  ☆76Updated 4 months ago
• commoncrawl / cc-webgraph
  Tools to construct and process webgraphs from Common Crawl data
  ☆87Updated this week
• simonw / llm-sentence-transformers
  LLM plugin for embeddings using sentence-transformers
  ☆52Updated last month
• urchade / GraphER
  GraphER: A Structure-aware Text-to-Graph Model for Entity and Relation Extraction
  ☆67Updated 7 months ago
• kenhktsui / anyclassifier
  One Line To Build Zero-Data Classifiers in Minutes
  ☆36Updated 5 months ago
• laura-ham / HM-Fashion-image-neural-search
  H&M Fashion Image similarity search with Weaviate and DocArray
  ☆42Updated 11 months ago