AI4Bharat / setu
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆12Updated 8 months ago
Alternatives and similar repositories for setu:
Users that are interested in setu are comparing it to the libraries listed below
- Solving data for LLMs - Create quality synthetic datasets!☆143Updated 3 months ago
- A blueprint for creating Pretraining and Fine-Tuning datasets for Indic languages☆96Updated 3 months ago
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆62Updated 2 months ago
- GenAI Experimentation☆57Updated last month
- Your local personalised AI agent☆41Updated last month
- Data extraction with LLM on CPU☆68Updated last year
- Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)☆73Updated this week
- Quick Notebook Tutorials☆29Updated 2 months ago
- Routing on Random Forest (RoRF)☆98Updated 3 months ago
- RAG example using DSPy, Gradio, FastAPI☆70Updated 9 months ago
- Low latency, High Accuracy, Custom Query routers for Humans and Agents. Built by Prithivi Da☆90Updated last month
- ☆44Updated 7 months ago
- A seamless matchmaking application that is programmed with Cohere Command R+, Stanford NLP DSPy framework, Weaviate Vector store and Crew…☆59Updated 8 months ago
- This repo is the central repo for all the RAG Evaluation reference material and partner workshop☆60Updated 3 months ago
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆32Updated 7 months ago
- This repository contains the code for dataset curation and finetuning of instruct variant of the Bilingual OpenHathi model. The resultin…☆23Updated last year
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆56Updated 2 months ago
- Dynamic Metadata based RAG Framework☆71Updated 5 months ago
- never forget anything again! combine AI and intelligent tooling for a local knowledge base to track catalogue, annotate, and plan for you…☆35Updated 8 months ago
- Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…☆48Updated 6 months ago
- AI real estate agent☆34Updated 11 months ago
- Dataset Viber is your chill repo for data collection, annotation and vibe checks.☆44Updated 4 months ago
- Mistral + Haystack: build RAG pipelines that rock 🤘☆100Updated 11 months ago
- A simple, consistent and extendable toolkit for IndicTrans2☆20Updated this week
- blablado is an extensible Assistant that listens to your voice and can execute custom Python functions you provided. It can speak as well…☆70Updated 5 months ago
- Code interpreter support for o1☆32Updated 4 months ago
- ☆63Updated 3 months ago
- Open source implementation of Sova - RAG-based Web search engine using power of LLMs. Using Langchain, Ollama, HuggingFace Embeddings and…☆33Updated 11 months ago
- OpenMindedChatbot is a Proof Of Concept that leverages the power of Open source Large Language Models (LLM) with Function Calling capabil…☆29Updated last year
- ☆59Updated 2 months ago