AI4Bharat / setu
Setu is a comprehensive pipeline designed to clean, filter, and deduplicate diverse data sources including Web, PDF, and Speech data. Built on Apache Spark, Setu encompasses four key stages: document preparation, document cleaning and analysis, flagging and filtering, and deduplication.
☆13Updated 9 months ago
Alternatives and similar repositories for setu:
Users that are interested in setu are comparing it to the libraries listed below
- A lightweight evaluation suite tailored specifically for assessing Indic LLMs across a diverse range of tasks☆32Updated 8 months ago
- Chat to your Database GenAI Chatbot☆31Updated 2 months ago
- ☆32Updated last month
- Dynamic Metadata based RAG Framework☆71Updated 6 months ago
- High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡☆64Updated 3 months ago
- Writing Blog Posts with Generative Feedback Loops!☆47Updated 11 months ago
- Explore a curated collection of exceptional open-source libraries for generative AI meticulously reviewed or slated for review by The AI …☆49Updated last year
- Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)☆76Updated last week
- ☆16Updated 8 months ago
- ☆19Updated 3 weeks ago
- Data extraction with LLM on CPU☆67Updated last year
- GenAI Experimentation☆57Updated 3 weeks ago
- Low latency, High Accuracy, Custom Query routers for Humans and Agents. Built by Prithivi Da☆93Updated 2 months ago
- Code for evaluating with Flow-Judge-v0.1 - an open-source, lightweight (3.8B) language model optimized for LLM system evaluations. Crafte…☆59Updated 3 months ago
- RAG with postgreSQL(nebius) and pgvector☆24Updated 2 months ago
- OpenAI GPT hosted Agent Framework for Windows and MacOS☆36Updated 7 months ago
- End-to-End LLM Guide☆101Updated 7 months ago
- AI real estate agent☆34Updated last year
- Solving data for LLMs - Create quality synthetic datasets!☆145Updated last month
- Own your AI, search the web with it🌐😎☆79Updated last month
- GPT-4 Level Conversational QA Trained In a Few Hours☆58Updated 6 months ago
- Your local personalised AI agent☆41Updated 2 months ago
- A tutorial on DSPy and whether automated prompt engineering lives up to the hype☆22Updated 9 months ago
- ☆1Updated 7 months ago
- never forget anything again! combine AI and intelligent tooling for a local knowledge base to track catalogue, annotate, and plan for you…☆37Updated 9 months ago
- DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.☆31Updated last year
- Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…☆48Updated 7 months ago
- A list of AI memory projects☆76Updated last month
- Mistral + Haystack: build RAG pipelines that rock 🤘☆100Updated last year