AI4Bharat / setuLinks
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆16Updated last year
Alternatives and similar repositories for setu
Users that are interested in setu are comparing it to the libraries listed below
Sorting:
- Data extraction with LLM on CPU☆68Updated last year
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆66Updated 9 months ago
- Quick Notebook Tutorials☆36Updated 3 weeks ago
- Solving data for LLMs - Create quality synthetic datasets!☆150Updated 6 months ago
- Official homepage for "Self-Harmonized Chain of Thought" (NAACL 2025)☆91Updated 6 months ago
- Writing Blog Posts with Generative Feedback Loops!☆50Updated last year
- ☆19Updated last year
- A personal knowledge base that I can dump information to and help me learn☆24Updated 2 months ago
- This repo is the central repo for all the RAG Evaluation reference material and partner workshop☆73Updated 3 months ago
- 💙 Unstructured Data Connectors for Haystack 2.0☆17Updated last year
- Verbosity control for AI agents☆64Updated last year
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆37Updated last year
- Simple Graph Memory for AI applications☆89Updated 2 months ago
- Recipes and resources for building, deploying, and fine-tuning generative AI with Fireworks.☆120Updated last week
- ☆54Updated 6 months ago
- Data extraction with LLM on CPU☆113Updated last year
- Diagnose the performance of your RAG🩺☆42Updated 4 months ago
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆76Updated 9 months ago
- 🚀 Scale your RAG pipeline using Ragswift: A scalable centralized embeddings management platform☆38Updated last year
- Dataset Viber is your chill repo for data collection, annotation and vibe checks.☆47Updated 11 months ago
- Medical Mixture of Experts LLM using Mergekit.☆20Updated last year
- RAG example using DSPy, Gradio, FastAPI☆83Updated last year
- Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models☆111Updated 3 months ago
- Simple customizable evaluation for text retrieval performance of Sentence Transformers embedders on PDFs☆27Updated 6 months ago
- Dynamic Metadata based RAG Framework☆75Updated last year
- A seamless matchmaking application that is programmed with Cohere Command R+, Stanford NLP DSPy framework, Weaviate Vector store and Crew…☆59Updated last year
- Mistral + Haystack: build RAG pipelines that rock 🤘☆105Updated last year
- Explore the use of DSPy for extracting features from PDFs 🔎☆45Updated last year
- A tutorial on DSPy and whether automated prompt engineering lives up to the hype☆24Updated last year
- ☆20Updated 9 months ago