AI4Bharat / setuLinks
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆16Updated last year
Alternatives and similar repositories for setu
Users that are interested in setu are comparing it to the libraries listed below
Sorting:
- Data extraction with LLM on CPU☆68Updated last year
- ☆30Updated last year
- Writing Blog Posts with Generative Feedback Loops!☆50Updated last year
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆67Updated 10 months ago
- ☆54Updated last month
- Mistral + Haystack: build RAG pipelines that rock 🤘☆105Updated last year
- Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…☆49Updated last year
- Recipes and resources for building, deploying, and fine-tuning generative AI with Fireworks.☆124Updated 2 weeks ago
- Solving data for LLMs - Create quality synthetic datasets!☆150Updated 8 months ago
- Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models☆111Updated 5 months ago
- Explore the use of DSPy for extracting features from PDFs 🔎☆45Updated last year
- Verbosity control for AI agents☆65Updated last year
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆78Updated 10 months ago
- Query language for blending SQL and LLMs across structured + unstructured data, with type constraints.☆113Updated this week
- Dataset Viber is your chill repo for data collection, annotation and vibe checks.☆46Updated last year
- This repo is the central repo for all the RAG Evaluation reference material and partner workshop☆76Updated 5 months ago
- ☆48Updated last year
- Data extraction with LLM on CPU☆112Updated last year
- Median is an open-source flashcard application that leverages the power of spaced repetition and artificial intelligence to transform the…☆22Updated 10 months ago
- Data extraction with LLM on CPU☆85Updated last year
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆38Updated last year
- RAG example using DSPy, Gradio, FastAPI☆85Updated last year
- Synthetic Data Generation using LLM via Argilla, Distilabel, ChatGPT, etc.☆30Updated last year
- Dynamic Metadata based RAG Framework☆75Updated last year
- Supervised instruction finetuning for LLM with HF trainer and Deepspeed☆35Updated 2 years ago
- Official homepage for "Self-Harmonized Chain of Thought" (NAACL 2025)☆92Updated 8 months ago
- Text to Python Objects via a LLM Function Call☆58Updated last year
- Your local personalised AI agent☆42Updated 10 months ago
- Simple Graph Memory for AI applications☆90Updated 4 months ago
- ☆210Updated 3 months ago