bminixhofer / tokenkitLinks
A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.
☆25Updated 3 weeks ago
Alternatives and similar repositories for tokenkit
Users that are interested in tokenkit are comparing it to the libraries listed below
Sorting:
- Code for Zero-Shot Tokenizer Transfer☆128Updated 4 months ago
- ☆57Updated 8 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆57Updated 9 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆56Updated 3 weeks ago
- ☆25Updated 4 months ago
- EvaByte: Efficient Byte-level Language Models at Scale☆101Updated last month
- This is the official repository for Inheritune.☆111Updated 3 months ago
- ☆58Updated 3 weeks ago
- Lightweight tools for quick and easy LLM demo's☆27Updated 8 months ago
- ☆55Updated this week
- An easy-to-understand framework for LLM samplers that rewind and revise generated tokens☆139Updated 3 months ago
- A framework for pitting LLMs against each other in an evolving library of games ⚔☆32Updated last month
- Lego for GRPO☆28Updated last week
- ☆34Updated 11 months ago
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆32Updated 3 months ago
- Supercharge huggingface transformers with model parallelism.☆77Updated 8 months ago
- An introduction to LLM Sampling☆78Updated 5 months ago
- Using open source LLMs to build synthetic datasets for direct preference optimization☆63Updated last year
- ☆45Updated 4 months ago
- ☆47Updated 9 months ago
- Codebase accompanying the Summary of a Haystack paper.☆78Updated 8 months ago
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆25Updated 6 months ago
- Repository containing the SPIN experiments on the DIBT 10k ranked prompts☆24Updated last year
- Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).☆80Updated last year
- 🚢 Data Toolkit for Sailor Language Models☆91Updated 3 months ago
- The first dense retrieval model that can be prompted like an LM☆73Updated 3 weeks ago
- Lightweight toolkit package to train and fine-tune 1.58bit Language models☆69Updated 2 weeks ago
- The official code repo and data hub of top_nsigma sampling strategy for LLMs.☆25Updated 3 months ago
- Experiments for efforts to train a new and improved t5☆77Updated last year
- Maya: An Instruction Finetuned Multilingual Multimodal Model using Aya☆110Updated 2 weeks ago