marta1994 / efficient_bpe_explanationLinks
This repository provides a clear, educational implementation of Byte Pair Encoding (BPE) tokenization in plain Python. The focus is on algorithmic understanding, not raw performance.
☆14Updated last year
Alternatives and similar repositories for efficient_bpe_explanation
Users that are interested in efficient_bpe_explanation are comparing it to the libraries listed below
Sorting:
- Interpretability for sequence generation models 🐛 🔍☆453Updated last week
- Llama from scratch, or How to implement a paper without crying☆584Updated last year
- 🦖 X—LLM: Cutting Edge & Easy LLM Finetuning☆408Updated 2 years ago
- Deep learning for dummies. All the practical details and useful utilities that go into working with real models.☆829Updated 6 months ago
- A set of scripts and notebooks on LLM finetunning and dataset creation☆116Updated last year
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆260Updated 2 years ago
- The repository for the code of the UltraFastBERT paper☆518Updated last year
- Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript☆616Updated last year
- List of papers on hallucination detection in LLMs.☆1,041Updated last month
- What would you do with 1000 H100s...☆1,151Updated 2 years ago
- 🤖 A PyTorch library of curated Transformer models and their composable components☆894Updated last year
- An open collection of implementation tips, tricks and resources for training large language models☆498Updated 2 years ago
- Stanford NLP Python library for understanding and improving PyTorch models via interventions☆857Updated 2 weeks ago
- Fast & Simple repository for pre-training and fine-tuning T5-style models☆1,019Updated last year
- Mass-editing thousands of facts into a transformer memory (ICLR 2023)☆538Updated 2 years ago
- Puzzles for exploring transformers☆386Updated 2 years ago
- Sparsify transformers with SAEs and transcoders☆692Updated this week
- Prune transformer layers☆74Updated last year
- Tools for understanding how transformer predictions are built layer-by-layer☆567Updated 6 months ago
- A list of awesome open source projects in the machine learning field, who's developers are mainly based in Germany☆52Updated last year
- LLM Workshop by Sourab Mangrulkar☆401Updated last year
- A collection of LogitsProcessors to customize and enhance LLM behavior for specific tasks.☆382Updated 7 months ago
- Best practices for distilling large language models.☆604Updated 2 years ago
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆294Updated 11 months ago
- Best practices & guides on how to write distributed pytorch training code☆576Updated 3 months ago
- Automatically split your PyTorch models on multiple GPUs for training & inference☆656Updated 2 years ago
- Chat Templates for 🤗 HuggingFace Large Language Models☆713Updated last year
- A framework for few-shot evaluation of autoregressive language models.☆13Updated last year
- Highly commented implementations of Transformers in PyTorch☆138Updated 2 years ago
- Easily embed, cluster and semantically label text datasets☆592Updated last year