AI4Bharat / setuLinks
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆15Updated last year
Alternatives and similar repositories for setu
Users that are interested in setu are comparing it to the libraries listed below
Sorting:
- Data extraction with LLM on CPU☆68Updated 2 years ago
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆68Updated last week
- ☆21Updated last year
- Verbosity control for AI agents☆64Updated last year
- Writing Blog Posts with Generative Feedback Loops!☆50Updated last year
- Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models☆113Updated 7 months ago
- Solving data for LLMs - Create quality synthetic datasets!☆150Updated 10 months ago
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆78Updated last year
- 💙 Unstructured Data Connectors for Haystack 2.0☆17Updated 2 years ago
- ☆31Updated 10 months ago
- Simple customizable evaluation for text retrieval performance of Sentence Transformers embedders on PDFs☆29Updated 10 months ago
- Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…☆51Updated last year
- Recipes and resources for building, deploying, and fine-tuning generative AI with Fireworks.☆129Updated last month
- Dynamic Metadata based RAG Framework☆78Updated last year
- Explore the use of DSPy for extracting features from PDFs 🔎☆49Updated last year
- LLM-Training-API: Including Embeddings & ReRankers, mergekit, LaserRMT☆27Updated last year
- Own your AI, search the web with it🌐😎☆92Updated 10 months ago
- Function Calling Benchmark & Testing☆92Updated last year
- Official homepage for "Self-Harmonized Chain of Thought" (NAACL 2025)☆91Updated 10 months ago
- Simple Graph Memory for AI applications☆89Updated 6 months ago
- Supervised instruction finetuning for LLM with HF trainer and Deepspeed☆36Updated 2 years ago
- Mistral + Haystack: build RAG pipelines that rock 🤘☆106Updated last year
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆38Updated last year
- A Python library to orchestrate LLMs in a neural network-inspired structure☆51Updated last year
- Synthetic Text Dataset Generation for LLM projects☆47Updated last week
- LLM prompt language based on Jinja. Banks provides tools and functions to build prompts text and chat messages from generic blueprints. I…☆117Updated 4 months ago
- ☆210Updated 5 months ago
- A personal knowledge base that I can dump information to and help me learn☆24Updated 6 months ago
- This repo is the central repo for all the RAG Evaluation reference material and partner workshop☆76Updated 7 months ago
- Repository for fine-tuning gemma models using unsloth for indic languages☆96Updated last year