An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.
☆61Jul 18, 2025Updated 7 months ago
Alternatives and similar repositories for qa_metrics
Users that are interested in qa_metrics are comparing it to the libraries listed below
Sorting:
- Reinforcement Learning of Vision Language Models with Self Visual Perception Reward☆161Sep 23, 2025Updated 5 months ago
- Official Implementation of "Learning to Refuse: Towards Mitigating Privacy Risks in LLMs"☆10Dec 13, 2024Updated last year
- Butler 是一个用于自动化服务管理和任务调度的工具项目。☆16Updated this week
- [NeurIPS 2024 D&B] Evaluating Copyright Takedown Methods for Language Models☆17Jul 17, 2024Updated last year
- ☆32Aug 9, 2024Updated last year
- Official repo for EMNLP'24 paper "SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning"☆29Oct 1, 2024Updated last year
- Official repository of the "Transformer Fusion with Optimal Transport" paper, published as a conference paper at ICLR 2024.☆31Apr 19, 2024Updated last year
- Converter for EN16931 invoices from CII to UBL☆39Feb 22, 2026Updated last week
- Sensitive-rs is a Rust library for finding, validating, filtering, and replacing sensitive words. It provides efficient algorithms to han…☆22Feb 4, 2026Updated last month
- [NeurIPS25] Official repo for "Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning"☆42Oct 3, 2025Updated 5 months ago
- [ICLR 2025] A Closer Look at Machine Unlearning for Large Language Models☆45Dec 4, 2024Updated last year
- ☆44Oct 1, 2024Updated last year
- ☆44Mar 3, 2023Updated 3 years ago
- Official repository for the paper "ALERT: A Comprehensive Benchmark for Assessing Large Language Models’ Safety through Red Teaming"☆56Sep 20, 2024Updated last year
- This is the official code for the paper "Vaccine: Perturbation-aware Alignment for Large Language Models" (NeurIPS2024)☆49Jan 15, 2026Updated last month
- Code and data for the FACTOR paper☆53Nov 15, 2023Updated 2 years ago
- “悟道”数据☆51Jul 5, 2021Updated 4 years ago
- A toolkit for building dense retrievers with deep language models.☆64Sep 24, 2021Updated 4 years ago
- PsyChat: A Client-Centric Dialogue System for Mental Health Support☆62Sep 4, 2024Updated last year
- ☆73Jul 15, 2024Updated last year
- TyDiP Multilingual Politeness dataset and code☆12Oct 15, 2023Updated 2 years ago
- Implementations of online merging optimizers proposed by Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment☆81Jun 19, 2024Updated last year
- ☆70Apr 14, 2023Updated 2 years ago
- Tutorial: Introduction to CrewAI☆80Jul 5, 2024Updated last year
- Dataset associated with "BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation" paper☆86Mar 2, 2021Updated 5 years ago
- ☆83Sep 14, 2024Updated last year
- Source code for Jordan Boyd-Graber's academic webpage.☆11Feb 25, 2026Updated last week
- [ACL 2024] CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling☆216May 21, 2025Updated 9 months ago
- Dataset of ML and NLP papers☆34Aug 17, 2022Updated 3 years ago
- RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024☆91Sep 30, 2024Updated last year
- Dataset for TACL 2022 paper: "FeTaQA: Free-form Table Question Answering"☆87May 11, 2023Updated 2 years ago
- Data Programming for Text Detection in Documents using SPEAR☆12Mar 26, 2025Updated 11 months ago
- [ICLR 2024]Data for "Multilingual Jailbreak Challenges in Large Language Models"☆99Mar 7, 2024Updated last year
- Knowledge-Based System'25☆11Dec 15, 2024Updated last year
- Open-source Human Feedback Library☆11Oct 25, 2023Updated 2 years ago
- Fine-tuning GPT-2 to generate research paper abstracts☆12Apr 28, 2021Updated 4 years ago
- TensorRT In Docker☆11Dec 7, 2024Updated last year
- Shiny apps for NGS etc based on reusable components created using Shiny modules☆88Dec 8, 2025Updated 2 months ago
- COMET for African languages☆10Jan 24, 2025Updated last year