Alternatives and similar repositories for tokenizer-analysis-suite

Users that are interested in tokenizer-analysis-suite are comparing it to the libraries listed below

• zouharvi / tokenization-scorer
  Simple-to-use scoring function for arbitrarily tokenized texts.
  ☆47Updated 9 months ago
• fdalvi / NeuroX
  A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.
  ☆105Updated 2 years ago
• ZurichNLP / mbr
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆60Updated last year
• huggingface / that_is_good_data
  ☆65Updated 2 years ago
• cisnlp / MEXA
  🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
  ☆11Updated 7 months ago
• MilaNLProc / simple-generation
  A python package to run inference with HuggingFace language and vision-language checkpoints wrapping many convenient features.
  ☆28Updated last year
• kanishkamisra / wugs-and-daxes
  Collection of academic works in natural language processing, computational linguistics, and computational cognitive science that study th…
  ☆22Updated last year
• JHU-CLSP / ettin-encoder-vs-decoder
  State-of-the-art paired encoder and decoder models (17M-1B params)
  ☆53Updated 3 months ago
• tylerachang / multilingual-geometry
  The geometry of multilingual language model representations (EMNLP 2022).
  ☆22Updated 3 years ago
• cshaib / diversity
  ☆21Updated 2 months ago
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆60Updated 2 weeks ago
• viking-sudo-rm / rusty-dawg
  Rust library for indexing and quickly searching large pretraining corpora
  ☆30Updated last month
• mainlp / germanic-lrl-corpora
  A survey of corpora for Germanic low-resource languages and dialects
  ☆26Updated 11 months ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆27Updated 11 months ago
• MilaNLProc / honest
  A Python package to compute HONEST, a score to measure hurtful sentence completions in language models. Published at NAACL 2021.
  ☆20Updated 7 months ago
• mohsenfayyaz / GlobEnc
  [NAACL 2022] GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers
  ☆21Updated 2 years ago
• machelreid / m2d2
  M2D2: A Massively Multi-domain Language Modeling Dataset (EMNLP 2022) by Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer
  ☆54Updated 3 years ago
• Kaleidophon / awesome-experimental-standards-deep-learning
  Repository collecting resources and best practices to improve experimental rigour in deep learning research.
  ☆27Updated 2 years ago
• g8a9 / ear
  Code associated with the paper "Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists"
  ☆50Updated 3 years ago
• mt-upc / transformer-contributions
  Measuring the Mixing of Contextual Information in the Transformer
  ☆33Updated 2 years ago
• jmerullo / lm_vector_arithmetic
  ☆36Updated 2 years ago
• nostalgebraist / transformer-utils
  Utilities for the HuggingFace transformers library
  ☆72Updated 2 years ago
• google-research / t5x_retrieval
  ☆101Updated 2 years ago
• SLAB-NLP / BUG
  A Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation, Levy et al., Findings of EMNLP 2021
  ☆14Updated 3 years ago
• google-research / url-nlp
  ☆220Updated 3 months ago
• kanishkamisra / minicons
  Utility for behavioral and representational analyses of Language Models
  ☆171Updated 2 months ago
• babylm / evaluation-pipeline-2024
  The evaluation pipeline for the 2024 BabyLM Challenge.
  ☆33Updated last year
• rewire-online / edos
  Public repository for SemEval 2023 - Task 10 - Explainable Detection of Online Sexism (EDOS)
  ☆25Updated 2 years ago
• google-deepmind / transformer_grammars
  Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale, TACL (2022)
  ☆133Updated 5 months ago
• alexandra-chron / hierarchical-domain-adaptation
  Code of NAACL 2022 "Efficient Hierarchical Domain Adaptation for Pretrained Language Models" paper.
  ☆32Updated 2 years ago