yixiaoer / tpu-training-exampleLinks
☆15Updated last year
Alternatives and similar repositories for tpu-training-example
Users that are interested in tpu-training-example are comparing it to the libraries listed below
Sorting:
- JAX implementation of the Mistral 7b v0.2 model☆36Updated last year
- A set of Python scripts that makes your experience on TPU better☆54Updated this week
- Transformer with Mu-Parameterization, implemented in Jax/Flax. Supports FSDP on TPU pods.☆32Updated 3 months ago
- Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.☆19Updated last month
- ☆89Updated last year
- Triton Implementation of HyperAttention Algorithm☆48Updated last year
- Experiment of using Tangent to autodiff triton☆81Updated last year
- Machine Learning eXperiment Utilities☆47Updated last month
- An implementation of the Llama architecture, to instruct and delight☆21Updated 3 months ago
- Automatically take good care of your preemptible TPUs☆36Updated 2 years ago
- ☆49Updated last year
- ☆46Updated last year
- Engineering the state of RNN language models (Mamba, RWKV, etc.)☆32Updated last year
- ☆82Updated last year
- ☆20Updated 2 years ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆60Updated 11 months ago
- some common Huggingface transformers in maximal update parametrization (µP)☆82Updated 3 years ago
- Experiments for efforts to train a new and improved t5☆76Updated last year
- ☆53Updated last year
- Collection of autoregressive model implementation☆86Updated 4 months ago
- Simple and efficient pytorch-native transformer training and inference (batched)☆78Updated last year
- ☆19Updated 4 months ago
- Using FlexAttention to compute attention with different masking patterns☆44Updated last year
- Unofficial but Efficient Implementation of "Mamba: Linear-Time Sequence Modeling with Selective State Spaces" in JAX☆88Updated last year
- Minimal (400 LOC) implementation Maximum (multi-node, FSDP) GPT training☆132Updated last year
- ☆85Updated last year
- Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"☆83Updated 10 months ago
- A basic pure pytorch implementation of flash attention☆16Updated 10 months ago
- ☆119Updated last year
- Here we will test various linear attention designs.☆62Updated last year