kuprel / minbpe-pytorch
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization, with PyTorch/CUDA
☆35Updated 11 months ago
Alternatives and similar repositories for minbpe-pytorch:
Users that are interested in minbpe-pytorch are comparing it to the libraries listed below
- ☆37Updated 6 months ago
- [WIP] Transformer to embed Danbooru labelsets☆13Updated 10 months ago
- Training hybrid models for dummies.☆18Updated 2 weeks ago
- Lightweight tools for quick and easy LLM demo's☆26Updated 4 months ago
- gzip Predicts Data-dependent Scaling Laws☆33Updated 8 months ago
- Utilities for Training Very Large Models☆57Updated 4 months ago
- ☆44Updated 6 months ago
- Because it's there.☆14Updated 4 months ago
- NLP with Rust for Python 🦀🐍☆60Updated 7 months ago
- an implementation of Self-Extend, to expand the context window via grouped attention☆118Updated last year
- GPU accelerated client-side embeddings for vector search, RAG etc.☆66Updated last year
- ☆62Updated 4 months ago
- Latent Large Language Models☆17Updated 5 months ago
- RWKV-7: Surpassing GPT☆73Updated 2 months ago
- An example implementation of RLHF (or, more accurately, RLAIF) built on MLX and HuggingFace.☆22Updated 7 months ago
- ☆36Updated 2 years ago
- The official evaluation suite and dynamic data release for MixEval.☆10Updated 4 months ago
- Standalone commandline CLI tool for compiling Triton kernels☆17Updated 4 months ago
- ☆52Updated 9 months ago
- ANE accelerated embedding models!☆15Updated last month
- Trying to deconstruct RWKV in understandable terms☆14Updated last year
- A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.☆16Updated 3 months ago
- ☆49Updated 10 months ago
- Github repo for Peifeng's internship project☆13Updated last year
- Super-fast Structured Outputs☆79Updated this week
- Efficiently computing & storing token n-grams from large corpora☆17Updated 3 months ago
- assign color hues to a collection of text fragments based on embeddings☆20Updated 7 months ago
- Efficient Dictionary Learning with Switch Sparse Autoencoders (SAEs)☆20Updated last month
- Tree-based indexes for neural-search☆29Updated 10 months ago