Alternatives and similar repositories for evals:

Users that are interested in evals are comparing it to the libraries listed below

• EleutherAI / elk
  Keeping language models honest by directly eliciting knowledge encoded in their activations.
  ☆193Updated this week
• google-research / cascades
  Python library which enables complex compositions of language models such as scratchpads, chain of thought, tool use, selection-inference…
  ☆200Updated 2 weeks ago
• meg-tong / sycophancy-eval
  datasets from the paper "Towards Understanding Sycophancy in Language Models"
  ☆69Updated last year
• justinchiu / openlogprobs
  Extract full next-token probabilities via language model APIs
  ☆229Updated 11 months ago
• anthropics / ConstitutionalHarmlessnessPaper
  ☆227Updated 2 years ago
• moirage / alignment-research-dataset
  A dataset of alignment research and code to reproduce it
  ☆73Updated last year
• aypan17 / machiavelli
  ☆128Updated 2 months ago
• kmeng01 / memit
  Mass-editing thousands of facts into a transformer memory (ICLR 2023)
  ☆460Updated 11 months ago
• ArthurConmy / Automatic-Circuit-Discovery
  ☆202Updated 3 months ago
• TransformerLensOrg / CircuitsVis
  Mechanistic Interpretability Visualizations using React
  ☆223Updated last month
• collin-burns / discovering_latent_knowledge
  ☆261Updated 10 months ago
• kaistAI / FLASK
  [ICLR 2024 Spotlight] FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
  ☆214Updated last year
• ofirpress / self-ask
  Code and data for "Measuring and Narrowing the Compositionality Gap in Language Models"
  ☆309Updated last year
• CarperAI / cheese
  Used for adaptive human in the loop evaluation of language and embedding models.
  ☆306Updated last year
• TomFrederik / unseal
  Mechanistic Interpretability for Transformer Models
  ☆49Updated 2 years ago
• METR / task-standard
  METR Task Standard
  ☆137Updated 3 weeks ago
• EleutherAI / concept-erasure
  Erasing concepts from neural representations with provable guarantees
  ☆221Updated this week
• steering-vectors / steering-vectors
  Steering vectors for transformer language models in Pytorch / Huggingface
  ☆81Updated 2 months ago
• lukasberglund / reversal_curse
  ☆280Updated last year
• nrimsky / CAA
  Steering Llama 2 with Contrastive Activation Addition
  ☆119Updated 8 months ago
• LoryPack / LLM-LieDetector
  Code for the ICLR 2024 paper "How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions"
  ☆64Updated 7 months ago
• shuyanzhou / docprompting
  Data and code for "DocPrompting: Generating Code by Retrieving the Docs" @ICLR 2023
  ☆236Updated last year
• orhonovich / unnatural-instructions
  ☆177Updated last year
• likenneth / othello_world
  Emergent world representations: Exploring a sequence model trained on a synthetic task
  ☆174Updated last year
• nrimsky / LM-exp
  LLM experiments done during SERI MATS - focusing on activation steering / interpreting activation spaces
  ☆84Updated last year
• METR / ai-rd-tasks
  ☆55Updated this week
• GraySwanAI / circuit-breakers
  Improving Alignment and Robustness with Circuit Breakers
  ☆176Updated 4 months ago
• neelnanda-io / 1L-Sparse-Autoencoder
  ☆117Updated last year
• CarperAI / autocrit
  A repository for transformer critique learning and generation
  ☆88Updated last year
• benlipkin / probsem
  Probabilistic LLM evaluations. [CogSci2023; ACL2023]
  ☆73Updated 6 months ago