AI4Bharat / setuLinks
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆15Updated last year
Alternatives and similar repositories for setu
Users that are interested in setu are comparing it to the libraries listed below
Sorting:
- Data extraction with LLM on CPU☆68Updated 2 years ago
- Writing Blog Posts with Generative Feedback Loops!☆50Updated last year
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆69Updated last month
- Simple Graph Memory for AI applications☆90Updated 7 months ago
- This repo is the central repo for all the RAG Evaluation reference material and partner workshop☆78Updated 8 months ago
- DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.☆43Updated last year
- A personal knowledge base that I can dump information to and help me learn☆24Updated 7 months ago
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆81Updated last year
- LLM-Training-API: Including Embeddings & ReRankers, mergekit, LaserRMT☆27Updated last year
- ☆30Updated last year
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆38Updated last year
- Own your AI, search the web with it🌐😎☆94Updated 11 months ago
- 🚀 Scale your RAG pipeline using Ragswift: A scalable centralized embeddings management platform☆38Updated last year
- This repo is for handling Question Answering, especially for Multi-hop Question Answering☆68Updated 2 years ago
- Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…☆51Updated last year
- Repository for fine-tuning gemma models using unsloth for indic languages☆97Updated last year
- ☆21Updated last year
- AI real estate agent☆36Updated last year
- Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models☆115Updated 9 months ago
- An advanced distributed knowledge fabric for intelligent document processing, featuring multi-document agents, optimized query handling, …☆48Updated 2 months ago
- Verbosity control for AI agents☆66Updated last year
- Your local personalised AI agent☆42Updated last year
- Data extraction with LLM on CPU☆112Updated 2 years ago
- Agentic RAG to help you build a startup🚀☆55Updated 9 months ago
- OpenMindedChatbot is a Proof Of Concept that leverages the power of Open source Large Language Models (LLM) with Function Calling capabil…☆30Updated 2 years ago
- Median is an open-source flashcard application that leverages the power of spaced repetition and artificial intelligence to transform the…☆22Updated last year
- Explore the use of DSPy for extracting features from PDFs 🔎☆49Updated last year
- Solving data for LLMs - Create quality synthetic datasets!☆151Updated 11 months ago
- Example implementation of Iteration of Tought - Gives a star if you like the project☆41Updated last year
- ☆55Updated 4 months ago