Alternatives and similar repositories for DS-1000

Users that are interested in DS-1000 are comparing it to the libraries listed below

• crux-eval / eval-arena

  View on GitHub
  ☆33Feb 2, 2026Updated 2 weeks ago
• zorazrw / odex

  View on GitHub
  [EMNLP'23] Execution-Based Evaluation for Open Domain Code Generation
  ☆49Dec 22, 2023Updated 2 years ago
• bigcode-project / bigcode-evaluation-harness

  View on GitHub
  A framework for the evaluation of autoregressive code generation language models.
  ☆1,020Jul 22, 2025Updated 6 months ago
• evalplus / evalplus

  View on GitHub
  Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024
  ☆1,687Oct 2, 2025Updated 4 months ago
• Timothyxxx / LMsMBTI

  View on GitHub
  A MBTI test on Large Language Model like GPT-3.
  ☆27May 2, 2022Updated 3 years ago
• facebookresearch / cruxeval

  View on GitHub
  CRUXEval: Code Reasoning, Understanding, and Execution Evaluation
  ☆165Oct 11, 2024Updated last year
• facebookresearch / mbr-exec

  View on GitHub
  code for "Natural Language to Code Translation with Execution"
  ☆41Nov 2, 2022Updated 3 years ago
• bigcode-project / bigcodebench

  View on GitHub
  [ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
  ☆478Jan 3, 2026Updated last month
• xlang-ai / Binder

  View on GitHub
  [ICLR 2023] Code for the paper "Binding Language Models in Symbolic Languages"
  ☆325Aug 25, 2023Updated 2 years ago
• rajasagashe / JuICe

  View on GitHub
  Code for generating the JuICe dataset.
  ☆37Oct 27, 2021Updated 4 years ago
• xlang-ai / UnifiedSKG

  View on GitHub
  [EMNLP 2022] Unifying and multi-tasking structured knowledge grounding with language models
  ☆568Aug 22, 2023Updated 2 years ago
• bigcode-project / octopack

  View on GitHub
  🐙 OctoPack: Instruction Tuning Code Large Language Models
  ☆478Feb 5, 2025Updated last year
• xlang-ai / icl-selective-annotation

  View on GitHub
  [ICLR 2023] Code for our paper "Selective Annotation Makes Language Models Better Few-Shot Learners"
  ☆109Jul 15, 2023Updated 2 years ago
• hendrycks / apps

  View on GitHub
  APPS: Automated Programming Progress Standard (NeurIPS 2021)
  ☆501Jun 19, 2024Updated last year
• Timothyxxx / EnvInteractiveLMPapers

  View on GitHub
  Paper collections of methods that using language to interact with environment, including interact with real world, simulated world or WWW…
  ☆129Jul 26, 2023Updated 2 years ago
• google-research / arcade-nl2code

  View on GitHub
  ☆54Aug 25, 2023Updated 2 years ago
• openai / human-eval

  View on GitHub
  Code for the paper "Evaluating Large Language Models Trained on Code"
  ☆3,127Jan 17, 2025Updated last year
• Timothyxxx / NeuralSymbolicPapers

  View on GitHub
  ☆15Aug 18, 2022Updated 3 years ago
• microsoft / DataScienceProblems

  View on GitHub
  A repository containing the Jupyter notebook code generation benchmark.
  ☆59Feb 9, 2022Updated 4 years ago
• 3B-Group / ConvRe

  View on GitHub
  🤖ConvRe🤯: An Investigation of LLMs’ Inefficacy in Understanding Converse Relations (EMNLP 2023)
  ☆24Oct 10, 2023Updated 2 years ago
• OpenLemur / Lemur

  View on GitHub
  [ICLR 2024] Lemur: Open Foundation Models for Language Agents
  ☆555Oct 28, 2023Updated 2 years ago
• zorazrw / multilingual-conala

  View on GitHub
  [EACL'23] MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
  ☆23Feb 13, 2023Updated 3 years ago
• sunlab-osu / CliniQG4QA

  View on GitHub
  CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering
  ☆23Feb 26, 2021Updated 4 years ago
• wangdeze18 / Multilingual-Adapter-for-SE

  View on GitHub
  ☆17Dec 9, 2022Updated 3 years ago
• nuprl / MultiPL-E

  View on GitHub
  A multi-programming language benchmark for LLMs
  ☆299Jan 28, 2026Updated 2 weeks ago
• sriniiyer / concode

  View on GitHub
  Mapping Language to Code in a Programmatic Context
  ☆80Jan 27, 2021Updated 5 years ago
• amazon-science / cceval

  View on GitHub
  CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS 2023)
  ☆172Aug 15, 2025Updated 6 months ago
• Victorwz / VaLM

  View on GitHub
  VaLM: Visually-augmented Language Modeling. ICLR 2023.
  ☆56Mar 6, 2023Updated 2 years ago
• GAIR-NLP / OlympicArena

  View on GitHub
  [NeurIPS 2024] OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
  ☆107Mar 6, 2025Updated 11 months ago
• microsoft / CodeT

  View on GitHub
  ☆671Nov 1, 2024Updated last year
• allenai / reward-bench

  View on GitHub
  RewardBench: the first evaluation tool for reward models.
  ☆687Jan 31, 2026Updated 2 weeks ago
• amazon-science / recode

  View on GitHub
  Releasing code for "ReCode: Robustness Evaluation of Code Generation Models"
  ☆58Mar 20, 2024Updated last year
• suzgunmirac / BIG-Bench-Hard

  View on GitHub
  Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
  ☆548Jun 25, 2024Updated last year
• GAIR-NLP / DatasetResearch

  View on GitHub
  DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discovery
  ☆20Sep 24, 2025Updated 4 months ago
• TIGER-AI-Lab / AceCoder

  View on GitHub
  The official repo for "AceCoder: Acing Coder RL via Automated Test-Case Synthesis" [ACL25]
  ☆96Apr 9, 2025Updated 10 months ago
• seketeam / DevEval

  View on GitHub
  A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
  ☆36Sep 4, 2024Updated last year
• CoderEval / CoderEval

  View on GitHub
  A collection of practical code generation tasks and tests in open source projects. Complementary to HumanEval by OpenAI.
  ☆154Dec 25, 2024Updated last year
• allenai / ask4help

  View on GitHub
  Code for the Ask4Help project
  ☆22Nov 24, 2022Updated 3 years ago
• microsoft / CodeXGLUE

  View on GitHub
  CodeXGLUE
  ☆1,801Apr 23, 2024Updated last year