Alternatives and similar repositories for tokenizer-analysis-suite

Users that are interested in tokenizer-analysis-suite are comparing it to the libraries listed below

• zouharvi / tokenization-scorer
  Simple-to-use scoring function for arbitrarily tokenized texts.
  ☆46Updated 10 months ago
• ZurichNLP / mbr
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆60Updated last year
• huggingface / that_is_good_data
  ☆65Updated 2 years ago
• fdalvi / NeuroX
  A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.
  ☆106Updated 2 years ago
• viking-sudo-rm / rusty-dawg
  Rust library for indexing and quickly searching large pretraining corpora
  ☆30Updated 2 months ago
• MilaNLProc / simple-generation
  A python package to run inference with HuggingFace language and vision-language checkpoints wrapping many convenient features.
  ☆28Updated last year
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆62Updated last week
• ltgoslo / ltg-bert
  LTG-Bert
  ☆34Updated 2 years ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆27Updated last year
• zouharvi / subset2evaluate
  Find informative examples to efficiently (human)-evaluate NLG models.
  ☆17Updated last month
• JHU-CLSP / ettin-encoder-vs-decoder
  State-of-the-art paired encoder and decoder models (17M-1B params)
  ☆54Updated 5 months ago
• nostalgebraist / transformer-utils
  Utilities for the HuggingFace transformers library
  ☆73Updated 2 years ago
• tylerachang / multilingual-geometry
  The geometry of multilingual language model representations (EMNLP 2022).
  ☆22Updated 3 years ago
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆64Updated last year
• cshaib / diversity
  ☆22Updated 3 months ago
• cisnlp / Glot500
  Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023
  ☆107Updated last year
• justinchiu / openlogprobs
  Extract full next-token probabilities via language model APIs
  ☆248Updated last year
• kanishkamisra / minicons
  Utility for behavioral and representational analyses of Language Models
  ☆172Updated 2 weeks ago
• aryamanarora / causalgym
  CausalGym: Benchmarking causal interpretability methods on linguistic tasks
  ☆51Updated last year
• jmerullo / lm_vector_arithmetic
  ☆36Updated 2 years ago
• allenai / wimbd
  What's In My Big Data (WIMBD) - a toolkit for analyzing large text datasets
  ☆225Updated last year
• mohsenfayyaz / GlobEnc
  [NAACL 2022] GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers
  ☆21Updated 2 years ago
• mainlp / germanic-lrl-corpora
  A survey of corpora for Germanic low-resource languages and dialects
  ☆26Updated last year
• google-research / url-nlp
  ☆231Updated 5 months ago
• konstantinjdobler / focus
  [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"
  ☆35Updated 7 months ago
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆142Updated 11 months ago
• machelreid / m2d2
  M2D2: A Massively Multi-domain Language Modeling Dataset (EMNLP 2022) by Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer
  ☆54Updated 3 years ago
• google-research / t5x_retrieval
  ☆102Updated 3 years ago
• ARBORproject / arborproject.github.io
  ☆83Updated 10 months ago
• cisnlp / MEXA
  🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
  ☆11Updated 9 months ago