Alternatives and similar repositories for evals:

Users that are interested in evals are comparing it to the libraries listed below

• EleutherAI / elk
  Keeping language models honest by directly eliciting knowledge encoded in their activations.
  ☆197Updated this week
• meg-tong / sycophancy-eval
  datasets from the paper "Towards Understanding Sycophancy in Language Models"
  ☆73Updated last year
• TransformerLensOrg / CircuitsVis
  Mechanistic Interpretability Visualizations using React
  ☆233Updated 3 months ago
• aypan17 / machiavelli
  ☆130Updated 4 months ago
• ArthurConmy / Automatic-Circuit-Discovery
  ☆211Updated 5 months ago
• anthropics / ConstitutionalHarmlessnessPaper
  ☆232Updated 2 years ago
• ofirpress / self-ask
  Code and data for "Measuring and Narrowing the Compositionality Gap in Language Models"
  ☆310Updated last year
• kmeng01 / memit
  Mass-editing thousands of facts into a transformer memory (ICLR 2023)
  ☆470Updated last year
• kaistAI / FLASK
  [ICLR 2024 Spotlight] FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
  ☆214Updated last year
• princeton-nlp / intercode
  [NeurIPS 2023 D&B] Code repository for InterCode benchmark https://arxiv.org/abs/2306.14898
  ☆209Updated 10 months ago
• EleutherAI / concept-erasure
  Erasing concepts from neural representations with provable guarantees
  ☆226Updated last month
• neelnanda-io / 1L-Sparse-Autoencoder
  ☆121Updated last year
• CarperAI / cheese
  Used for adaptive human in the loop evaluation of language and embedding models.
  ☆306Updated 2 years ago
• google-research / cascades
  Python library which enables complex compositions of language models such as scratchpads, chain of thought, tool use, selection-inference…
  ☆204Updated 2 months ago
• steering-vectors / steering-vectors
  Steering vectors for transformer language models in Pytorch / Huggingface
  ☆90Updated last month
• justinchiu / openlogprobs
  Extract full next-token probabilities via language model APIs
  ☆233Updated last year
• moirage / alignment-research-dataset
  A dataset of alignment research and code to reproduce it
  ☆74Updated last year
• CarperAI / autocrit
  A repository for transformer critique learning and generation
  ☆89Updated last year
• evandez / REMEDI
  Inspecting and Editing Knowledge Representations in Language Models
  ☆112Updated last year
• collin-burns / discovering_latent_knowledge
  ☆262Updated last year
• tianjunz / HIR
  ☆159Updated 2 years ago
• tomekkorbak / pretraining-with-human-feedback
  Code accompanying the paper Pretraining Language Models with Human Preferences
  ☆180Updated last year
• normster / llm_rules
  RuLES: a benchmark for evaluating rule-following in language models
  ☆220Updated 3 weeks ago
• UKGovernmentBEIS / inspect_evals
  Collection of evals for Inspect AI
  ☆92Updated this week
• anthropics / toy-models-of-superposition
  Notebooks accompanying Anthropic's "Toy Models of Superposition" paper
  ☆116Updated 2 years ago
• nrimsky / LM-exp
  LLM experiments done during SERI MATS - focusing on activation steering / interpreting activation spaces
  ☆91Updated last year
• likenneth / othello_world
  Emergent world representations: Exploring a sequence model trained on a synthetic task
  ☆177Updated last year
• AlignmentResearch / tuned-lens
  Tools for understanding how transformer predictions are built layer-by-layer
  ☆479Updated 9 months ago
• reasoning-machines / prompt-lib
  A set of utilities for running few-shot prompting experiments on large-language models
  ☆118Updated last year