Alternatives and similar repositories for tokenizer-analysis-suite

Users that are interested in tokenizer-analysis-suite are comparing it to the libraries listed below

• ZurichNLP / mbr
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆60Updated last year
• zouharvi / subset2evaluate
  Find informative examples to efficiently (human)-evaluate NLG models.
  ☆16Updated this week
• SLAB-NLP / BUG
  A Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation, Levy et al., Findings of EMNLP 2021
  ☆14Updated 3 years ago
• alexa / ramen
  A software for transferring pre-trained English models to foreign languages
  ☆19Updated 2 years ago
• tylerachang / multilingual-geometry
  The geometry of multilingual language model representations (EMNLP 2022).
  ☆21Updated 2 years ago
• fdalvi / NeuroX
  A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.
  ☆106Updated 2 years ago
• zouharvi / tokenization-scorer
  Simple-to-use scoring function for arbitrarily tokenized texts.
  ☆46Updated 7 months ago
• mt-upc / transformer-contributions
  Measuring the Mixing of Contextual Information in the Transformer
  ☆31Updated 2 years ago
• cshaib / diversity
  ☆20Updated 2 weeks ago
• katherinethai / par3
  ☆29Updated 10 months ago
• MadryLab / AT2
  Attribute statements generated by LLMs to preceding tokens using attention weights.
  ☆17Updated 5 months ago
• MilaNLProc / simple-generation
  A python package to run inference with HuggingFace language and vision-language checkpoints wrapping many convenient features.
  ☆28Updated last year
• babylm / evaluation-pipeline-2024
  The evaluation pipeline for the 2024 BabyLM Challenge.
  ☆33Updated 10 months ago
• huggingface / that_is_good_data
  ☆65Updated 2 years ago
• rewire-online / edos
  Public repository for SemEval 2023 - Task 10 - Explainable Detection of Online Sexism (EDOS)
  ☆23Updated 2 years ago
• mt-upc / transformer-contributions-nmt
  ☆17Updated 3 years ago
• cisnlp / MEXA
  🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
  ☆11Updated 6 months ago
• interpretingdl / eacl2024_transformer_interpretability_tutorial
  Materials for EACL2024 tutorial: Transformer-specific Interpretability
  ☆60Updated last year
• rudinger / winogender-schemas
  Data for evaluating gender bias in coreference resolution systems.
  ☆80Updated 6 years ago
• MaLA-LM / GlotEval
  GlotEval: a unified evaluation toolkit designed to benchmark multilingual Large Language Models (LLMs) in a language-specific way
  ☆14Updated 2 months ago
• mainlp / germanic-lrl-corpora
  A survey of corpora for Germanic low-resource languages and dialects
  ☆25Updated 10 months ago
• cisnlp / Glot500
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆104Updated last year
• PAIR-code / interpretability
  PAIR.withgoogle.com and friend's work on interpretability methods
  ☆203Updated 2 weeks ago
• mohsenfayyaz / DecompX
  DecompX: Explaining Transformers Decisions by Propagating Token Decomposition [ACL 2023]
  ☆18Updated 3 months ago
• langtech-bsc / mt-evaluation
  A framework for evaluating Machine Translation models.
  ☆10Updated 4 months ago
• marzenakrp / demetr
  Repository for DEMETR: Diagnosing Evaluation Metrics for Translation
  ☆15Updated 2 years ago
• ARBORproject / arborproject.github.io
  ☆81Updated 7 months ago
• zouharvi / ryanize-bib
  Highlight errors in a bib file: missing URLs, capitalization protection, etc
  ☆27Updated last year
• Kaleidophon / awesome-experimental-standards-deep-learning
  Repository collecting resources and best practices to improve experimental rigour in deep learning research.
  ☆27Updated 2 years ago
• viking-sudo-rm / rusty-dawg
  Rust library for indexing and quickly searching large pretraining corpora
  ☆29Updated this week