bminixhofer / tokenkitLinks
A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.
☆44Updated last month
Alternatives and similar repositories for tokenkit
Users that are interested in tokenkit are comparing it to the libraries listed below
Sorting:
- Code for Zero-Shot Tokenizer Transfer☆135Updated 7 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆63Updated 3 weeks ago
- ☆48Updated last year
- ☆57Updated 11 months ago
- ☆54Updated 9 months ago
- A fast implementation of T5/UL2 in PyTorch using Flash Attention☆107Updated 5 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆60Updated last year
- Repository for "I am a Strange Dataset: Metalinguistic Tests for Language Models"☆44Updated last year
- ☆38Updated last year
- 🕸 GlotCC Dataset and Pipline -- NeurIPS 2024☆20Updated 4 months ago
- ☆39Updated last year
- Synthetic Data Generation for Evaluation☆14Updated 6 months ago
- ☆49Updated 6 months ago
- ☆51Updated 7 months ago
- ☆78Updated last week
- 🚢 Data Toolkit for Sailor Language Models☆94Updated 6 months ago
- Reference implementation for Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model☆44Updated last year
- Supercharge huggingface transformers with model parallelism.☆77Updated last month
- ☆81Updated 2 months ago
- State-of-the-art paired encoder and decoder models (17M-1B params)☆44Updated 3 weeks ago
- This is the official repository for Inheritune.☆112Updated 6 months ago
- Repository containing the SPIN experiments on the DIBT 10k ranked prompts☆24Updated last year
- ☆58Updated 3 months ago
- ☆52Updated 10 months ago
- Efficient encoder-decoder architecture for small language models (≤1B parameters) with cross-architecture knowledge distillation and visi…☆29Updated 6 months ago
- Using open source LLMs to build synthetic datasets for direct preference optimization☆65Updated last year
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆24Updated 9 months ago
- [ICLR 2025] Monet: Mixture of Monosemantic Experts for Transformers☆70Updated 2 months ago
- Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).☆80Updated last year