☆59Nov 18, 2025Updated 3 months ago
Alternatives and similar repositories for trigrams
Users that are interested in trigrams are comparing it to the libraries listed below
Sorting:
- Code for SaGe subword tokenizer (EACL 2023)☆27Nov 30, 2024Updated last year
- ☆15Apr 2, 2025Updated 10 months ago
- A library for data streaming and augmentation☆21May 5, 2025Updated 9 months ago
- [NAACL 2024] Topics, Authors, and Institutions in Large Language Model Research: Trends from 17K arXiv Papers https://arxiv.org/abs/2307.…☆17Jan 27, 2024Updated 2 years ago
- Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"☆22Feb 14, 2024Updated 2 years ago
- ☆21Jul 1, 2021Updated 4 years ago
- 0-Shot Tokenizer Transplant☆14May 16, 2025Updated 9 months ago
- JAX Scalify: end-to-end scaled arithmetics☆18Oct 30, 2024Updated last year
- Label shift estimation for transfer difficulty with Familiarity.☆10Feb 4, 2025Updated last year
- HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation [ACL 2023]☆14Jul 11, 2023Updated 2 years ago
- Code for the examples presented in the talk "Training a Llama in your backyard: fine-tuning very large models on consumer hardware" given…☆15Oct 16, 2023Updated 2 years ago
- ☆12Jul 2, 2024Updated last year
- [Konvens21] This repository contains the DFKI MobIE Corpus, a dataset of 3,232 German-language documents that have been annotated with fi…☆12Sep 17, 2024Updated last year
- Lightweight piece tokenization library☆12Apr 15, 2024Updated last year
- Simple (fast) transformer inference in PyTorch with torch.compile + lit-llama code☆10Aug 29, 2023Updated 2 years ago
- Official implementation of the paper: "A deeper look at depth pruning of LLMs"☆15Jul 24, 2024Updated last year
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More☆34May 17, 2025Updated 9 months ago
- [WIP] Transformer to embed Danbooru labelsets☆13Mar 31, 2024Updated last year
- ☆35Apr 12, 2024Updated last year
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024)☆17Jun 18, 2024Updated last year
- The training codes of Jasper-Token-Compression-600M☆19Nov 19, 2025Updated 3 months ago
- ☆13Dec 17, 2021Updated 4 years ago
- German Language Understanding Evaluation Benchmark @NAACL24☆22Dec 11, 2025Updated 2 months ago
- Code for Zero-Shot Tokenizer Transfer☆143Jan 14, 2025Updated last year
- Official implementation of "GPT or BERT: why not both?"☆62Jul 28, 2025Updated 7 months ago
- PyTorch implementation of StableMask (ICML'24)☆15Jun 27, 2024Updated last year
- Minimal code to train ELMo models in recent versions of TensorFlow☆14Apr 30, 2023Updated 2 years ago
- ☆18Mar 18, 2024Updated last year
- ☆15Jun 14, 2024Updated last year
- [AAAI 2026] SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries☆37Jan 14, 2026Updated last month
- Highly specialized crate to parse and use `google/sentencepiece` 's precompiled_charsmap in `tokenizers`☆20Jan 8, 2026Updated last month
- ☆17Jun 11, 2025Updated 8 months ago
- DImensionality REduction in JAX☆25Nov 21, 2025Updated 3 months ago
- MEXMA: Token-level objectives improve sentence representations☆43Jan 6, 2025Updated last year
- c++ mosestokenizer☆18Mar 13, 2024Updated last year
- Repo for the paper: PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees (CVPR 2024)☆23Aug 14, 2024Updated last year
- ☆20May 30, 2024Updated last year
- Accelerate LLM preference tuning via prefix sharing with a single line of code☆51Jul 4, 2025Updated 7 months ago
- Flash Attention Triton kernel with support for second-order derivatives☆142Feb 4, 2026Updated 3 weeks ago