kuprel / minbpe-pytorchLinks
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA
☆40Updated last year
Alternatives and similar repositories for minbpe-pytorch
Users that are interested in minbpe-pytorch are comparing it to the libraries listed below
Sorting:
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆203Updated 2 months ago
- Inference Llama 2 in one file of zero-dependency, zero-unsafe Rust☆39Updated 2 years ago
- ☆132Updated last year
- Simple high-throughput inference library☆128Updated 4 months ago
- Inference of Mamba models in pure C☆192Updated last year
- an implementation of Self-Extend, to expand the context window via grouped attention☆118Updated last year
- Alice in Wonderland code base for experiments and raw experiments data☆131Updated last week
- Modular Rust transformer/LLM library using Candle☆37Updated last year
- Because it's there.☆16Updated last year
- [WIP] Transformer to embed Danbooru labelsets☆13Updated last year
- ☆20Updated 11 months ago
- Efficiently computing & storing token n-grams from large corpora☆26Updated 11 months ago
- WebGPU LLM inference tuned by hand☆151Updated 2 years ago
- Implementation of mamba with rust☆88Updated last year
- ☆35Updated 2 years ago
- GPU accelerated client-side embeddings for vector search, RAG etc.☆65Updated last year
- RWKV in nanoGPT style☆193Updated last year
- A library for incremental loading of large PyTorch checkpoints☆56Updated 2 years ago
- Fast and vectorizable algorithms for searching in a vector of sorted floating point numbers☆150Updated 9 months ago
- a small code base for training large models☆311Updated 5 months ago
- ☆157Updated 2 years ago
- NLP with Rust for Python 🦀🐍☆65Updated 4 months ago
- A sketch of a Transformer in Rust for a blog post☆32Updated 3 years ago
- Efficient vector database for hundred millions of embeddings.☆208Updated last year
- Lightweight tools for quick and easy LLM demo's☆28Updated last year
- Python bindings for ggml☆146Updated last year
- gzip Predicts Data-dependent Scaling Laws☆34Updated last year
- Full finetuning of large language models without large memory requirements☆94Updated this week
- This repository has code for fine-tuning LLMs with GRPO specifically for Rust Programming using cargo as feedback☆104Updated 6 months ago
- Experiments for efforts to train a new and improved t5☆76Updated last year