cimeister / tokenizer-analysis-suiteLinks
☆37Updated last month
Alternatives and similar repositories for tokenizer-analysis-suite
Users that are interested in tokenizer-analysis-suite are comparing it to the libraries listed below
Sorting:
- Find informative examples to efficiently (human)-evaluate NLG models.☆16Updated 3 weeks ago
- The evaluation pipeline for the 2024 BabyLM Challenge.☆33Updated 11 months ago
- The geometry of multilingual language model representations (EMNLP 2022).☆22Updated 3 years ago
- Minimum Bayes Risk Decoding for Hugging Face Transformers☆60Updated last year
- ☆20Updated last month
- German Language Understanding Evaluation Benchmark @NAACL24☆17Updated last month
- ☆65Updated 2 years ago
- A Python library that encapsulates various methods for neuron interpretation and analysis in Deep NLP models.☆106Updated 2 years ago
- GlotEval: a unified evaluation toolkit designed to benchmark multilingual Large Language Models (LLMs) in a language-specific way☆14Updated 3 months ago
- A software for transferring pre-trained English models to foreign languages☆19Updated 2 years ago
- Public repository for SemEval 2023 - Task 10 - Explainable Detection of Online Sexism (EDOS)☆24Updated 2 years ago
- [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"☆34Updated 4 months ago
- Utility for behavioral and representational analyses of Language Models☆165Updated last month
- A survey of corpora for Germanic low-resource languages and dialects☆25Updated 10 months ago
- DecompX: Explaining Transformers Decisions by Propagating Token Decomposition [ACL 2023]☆18Updated 3 months ago
- One-stop shop for running and fine-tuning transformer-based language models for retrieval☆59Updated last week
- Code for SaGe subword tokenizer (EACL 2023)☆27Updated 11 months ago
- Rust library for indexing and quickly searching large pretraining corpora☆29Updated this week
- Simple-to-use scoring function for arbitrarily tokenized texts.☆46Updated 8 months ago
- A python package to run inference with HuggingFace language and vision-language checkpoints wrapping many convenient features.☆28Updated last year
- Data for evaluating gender bias in coreference resolution systems.☆80Updated 6 years ago
- Measuring the Mixing of Contextual Information in the Transformer☆31Updated 2 years ago
- Simple and scalable tools for data-driven pretraining data selection.☆28Updated 4 months ago
- Repository collecting resources and best practices to improve experimental rigour in deep learning research.☆27Updated 2 years ago
- Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023☆106Updated last year
- Code associated with the paper "Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists"☆50Updated 3 years ago
- Utilities for the HuggingFace transformers library☆72Updated 2 years ago
- Interpretability for sequence generation models 🐛 🔍☆444Updated this week
- PAIR.withgoogle.com and friend's work on interpretability methods☆208Updated last month
- M2D2: A Massively Multi-domain Language Modeling Dataset (EMNLP 2022) by Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer☆54Updated 2 years ago