gautierdag / tokenizer-bench
Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"
☆10Updated 7 months ago
Related projects: ⓘ
- A library for data streaming and augmentation☆20Updated 6 months ago
- ☆19Updated last year
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆22Updated 5 months ago
- ☆13Updated 3 months ago
- This repo contains code for the paper "Psychologically-informed chain-of-thought prompts for metaphor understanding in large language mod…☆12Updated last year
- ☆19Updated last year
- ☆16Updated 10 months ago
- ☆11Updated 2 years ago
- LTG-Bert☆25Updated 8 months ago
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆42Updated 10 months ago
- Efficient interpolation-based ranking on CPUs☆10Updated 2 weeks ago
- Multilingual Open Text☆25Updated 5 months ago
- ☆13Updated this week
- Statistics on multilingual datasets☆17Updated 2 years ago
- Code for SaGe subword tokenizer (EACL 2023)☆21Updated this week
- Minimum Bayes Risk Decoding for Hugging Face Transformers☆51Updated 3 months ago
- M2D2: A Massively Multi-domain Language Modeling Dataset (EMNLP 2022) by Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer☆53Updated last year
- Library for fast text representation and classification.☆28Updated 8 months ago
- Embedding Recycling for Language models☆38Updated last year
- ☆46Updated 2 years ago
- Adding new tasks to T0 without catastrophic forgetting☆30Updated last year
- T-Projection is a method to perform high-quality Annotation Projection of Sequence Labeling datasets.☆11Updated 9 months ago
- ☆11Updated last year
- SeqScore: Scoring for named entity recognition and other sequence labeling tasks☆20Updated 10 months ago
- ☆15Updated last month
- HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation [ACL 2023]☆13Updated last year
- ☆20Updated 3 years ago
- Vocabulary Trimming (VT) is a model compression technique, which reduces a multilingual LM vocabulary to a target language by deleting ir…☆29Updated 3 weeks ago
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆33Updated last year
- Implementation of "SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages" paper, accepted to E…☆18Updated last year