marta1994 / efficient_bpe_explanationLinks
This repository provides a clear, educational implementation of Byte Pair Encoding (BPE) tokenization in plain Python. The focus is on algorithmic understanding, not raw performance.
☆13Updated last year
Alternatives and similar repositories for efficient_bpe_explanation
Users that are interested in efficient_bpe_explanation are comparing it to the libraries listed below
Sorting:
- Interpretability for sequence generation models 🐛 🔍☆451Updated this week
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆294Updated 10 months ago
- Best practices & guides on how to write distributed pytorch training code☆562Updated 2 months ago
- Best practices for distilling large language models.☆596Updated last year
- Llama from scratch, or How to implement a paper without crying☆582Updated last year
- A collection of LogitsProcessors to customize and enhance LLM behavior for specific tasks.☆380Updated 6 months ago
- 🦖 X—LLM: Cutting Edge & Easy LLM Finetuning☆407Updated last year
- ☆45Updated 7 months ago
- Deep learning for dummies. All the practical details and useful utilities that go into working with real models.☆829Updated 5 months ago
- Easily embed, cluster and semantically label text datasets☆587Updated last year
- Late Interaction Models Training & Retrieval☆679Updated this week
- TODa: Tamazight Open Dataset☆16Updated 11 months ago
- Prune transformer layers☆74Updated last year
- LLM Workshop by Sourab Mangrulkar☆400Updated last year
- Training Sparse Autoencoders on Language Models☆1,144Updated this week
- ☆414Updated this week
- Puzzles for exploring transformers☆382Updated 2 years ago
- awesome synthetic (text) datasets☆320Updated this week
- What would you do with 1000 H100s...☆1,143Updated 2 years ago
- Bringing BERT into modernity via both architecture changes and scaling☆1,607Updated 6 months ago
- Chat Templates for 🤗 HuggingFace Large Language Models☆711Updated last year
- The nnsight package enables interpreting and manipulating the internals of deep learned models.☆758Updated this week
- Fast bare-bones BPE for modern tokenizer training☆174Updated 6 months ago
- Stanford NLP Python library for understanding and improving PyTorch models via interventions☆849Updated 2 months ago
- Manage scalable open LLM inference endpoints in Slurm clusters☆278Updated last year
- ☆872Updated last month
- Well documented, unit tested, type checked and formatted implementation of a vanilla transformer - for educational purposes.☆273Updated last year
- A set of scripts and notebooks on LLM finetunning and dataset creation☆113Updated last year
- Sparsify transformers with SAEs and transcoders☆681Updated 2 weeks ago
- ☆559Updated last year