lucidrains / lion-pytorch
🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch
☆2,106Updated 3 months ago
Alternatives and similar repositories for lion-pytorch:
Users that are interested in lion-pytorch are comparing it to the libraries listed below
- Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models☆779Updated 8 months ago
- Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch☆651Updated 3 months ago
- Official codebase used to develop Vision Transformer, SigLIP, MLP-Mixer, LiT and more.☆2,735Updated this week
- Foundation Architecture for (M)LLMs☆3,062Updated 11 months ago
- A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.☆2,509Updated last month
- Structured state space sequence models☆2,581Updated 8 months ago
- TorchMultimodal is a PyTorch library for training state-of-the-art multimodal multi-task models at scale.☆1,561Updated this week
- A method to increase the speed and lower the memory footprint of existing vision transformers.☆1,025Updated 9 months ago
- A concise but complete full-attention transformer with a set of promising experimental features from various papers☆5,149Updated this week
- The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”☆955Updated last year
- torchview: visualize pytorch models☆892Updated last week
- Machine learning metrics for distributed, scalable PyTorch applications.☆2,228Updated this week
- D-Adaptation for SGD, Adam and AdaGrad☆518Updated 2 months ago
- Vector (and Scalar) Quantization, in Pytorch☆3,043Updated this week
- A simple way to keep track of an Exponential Moving Average (EMA) version of your Pytorch model☆565Updated 3 months ago
- Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch☆1,235Updated 2 years ago
- EVA Series: Visual Representation Fantasies from BAAI☆2,453Updated 7 months ago
- Schedule-Free Optimization in PyTorch☆2,116Updated 3 weeks ago
- Transformer based on a variant of attention that is linear complexity in respect to sequence length☆751Updated 10 months ago
- maximal update parametrization (µP)☆1,480Updated 8 months ago
- Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch☆1,132Updated last year
- solo-learn: a library of self-supervised methods for visual representation learning powered by Pytorch Lightning☆1,472Updated last month
- Tensors, for human consumption☆1,202Updated 4 months ago
- Cramming the training of a (BERT-type) language model into limited compute.☆1,324Updated 9 months ago
- Pytorch library for fast transformer implementations☆1,687Updated 2 years ago
- A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆25 knowledge distillation methods p…☆1,471Updated this week
- View model summaries in PyTorch!☆2,726Updated this week
- An implementation of "Retentive Network: A Successor to Transformer for Large Language Models"☆1,177Updated last year
- minLoRA: a minimal PyTorch library that allows you to apply LoRA to any PyTorch model.☆454Updated last year
- Hackable and optimized Transformers building blocks, supporting a composable construction.☆9,215Updated this week