bojone / tigerLinks
A Tight-fisted Optimizer
☆49Updated 2 years ago
Alternatives and similar repositories for tiger
Users that are interested in tiger are comparing it to the libraries listed below
Sorting:
- Lion and Adam optimization comparison☆63Updated 2 years ago
- A Tight-fisted Optimizer (Tiger), implemented in PyTorch.☆12Updated last year
- [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…☆102Updated last year
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆123Updated 7 months ago
- ☆106Updated last year
- Low-bit optimizers for PyTorch☆130Updated last year
- 基于Gated Attention Unit的Transformer模型(尝鲜版)☆98Updated 2 years ago
- Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models☆325Updated 6 months ago
- [EMNLP 2022] Official implementation of Transnormer in our EMNLP 2022 paper - The Devil in Linear Transformer☆62Updated 2 years ago
- This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…☆58Updated last year
- [EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling☆86Updated 2 years ago
- ☆14Updated last year
- Official implementation of TransNormerLLM: A Faster and Better LLM☆247Updated last year
- A MoE impl for PyTorch, [ATC'23] SmartMoE☆67Updated 2 years ago
- Implementation of "Attention Is Off By One" by Evan Miller☆196Updated 2 years ago
- qwen-nsa☆74Updated 4 months ago
- An Experiment on Dynamic NTK Scaling RoPE☆64Updated last year
- ☆82Updated 3 months ago
- ☆63Updated last year
- ☆197Updated last year
- A repository for DenseSSMs☆88Updated last year
- Research without Re-search: Maximal Update Parametrization Yields Accurate Loss Prediction across Scales☆32Updated 2 years ago
- [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models☆78Updated last year
- 32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.☆49Updated 2 years ago
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆136Updated last year
- [ICLR 2024] This is the official PyTorch implementation of "QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Mod…☆29Updated last year
- Models and examples built with OneFlow☆98Updated 10 months ago
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.☆98Updated last year
- ☆32Updated last year
- [ICML24] Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs☆92Updated 9 months ago