Qurrent-AI / RES-Q
RES-Q: Evaluating the Code-Editing Capability of Large Language Model Systems at the Repository Scale
☆26Updated 9 months ago
Alternatives and similar repositories for RES-Q:
Users that are interested in RES-Q are comparing it to the libraries listed below
- Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions☆41Updated 7 months ago
- ☆53Updated 6 months ago
- Vivaria is METR's tool for running evaluations and conducting agent elicitation research.☆85Updated this week
- ☆36Updated 2 months ago
- ☆124Updated last week
- Just a bunch of benchmark logs for different LLMs☆119Updated 8 months ago
- Contains random samples referenced in the paper "Sleeper Agents: Training Robustly Deceptive LLMs that Persist Through Safety Training".☆98Updated last year
- Scaling is a distributed training library and installable dependency designed to scale up neural networks, with a dedicated module for tr…☆58Updated 5 months ago
- ☆87Updated 2 weeks ago
- Verdict is a library for scaling judge-time compute.☆190Updated 2 weeks ago
- Releases from OpenAI Preparedness☆276Updated this week
- Sphynx Hallucination Induction☆53Updated 2 months ago
- Code for the paper "Fishing for Magikarp"☆151Updated 2 weeks ago
- ☆67Updated 2 months ago
- Code for NeurIPS'24 paper 'Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization'☆186Updated 4 months ago
- Evaluating LLMs with CommonGen-Lite☆89Updated last year
- r2e: turn any github repository into a programming agent environment☆108Updated last month
- Red-Teaming Language Models with DSPy☆175Updated last month
- SWE Arena☆28Updated this week
- A benchmark that challenges language models to code solutions for scientific problems☆111Updated last week
- an open source reproduction of NVIDIA's nGPT (Normalized Transformer with Representation Learning on the Hypersphere)☆91Updated 3 weeks ago
- Track the progress of LLM context utilisation☆54Updated 8 months ago
- CodeSage: Code Representation Learning At Scale (ICLR 2024)☆99Updated 5 months ago
- Functional Benchmarks and the Reasoning Gap☆84Updated 6 months ago
- Small, simple agent task environments for training and evaluation☆18Updated 5 months ago
- Open sourced predictions, execution logs, trajectories, and results from model inference + evaluation runs on the SWE-bench task.☆157Updated this week
- Train your own SOTA deductive reasoning model☆81Updated 3 weeks ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆55Updated 7 months ago
- ☆80Updated 2 months ago
- Mixing Language Models with Self-Verification and Meta-Verification☆102Updated 3 months ago