Alternatives and similar repositories for Scalable-Softmax

Users that are interested in Scalable-Softmax are comparing it to the libraries listed below

• GATECH-EIC / SuperTickets
  [ECCV 2022] SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning
  ☆20Updated 3 years ago
• softmax1 / Flash-Attention-Softmax-N
  CUDA and Triton implementations of Flash Attention with SoftmaxN.
  ☆73Updated last year
• TianjinYellow / StableSPAM
  ☆27Updated 9 months ago
• GindaChen / FlexFlashAttention3
  FlexAttention w/ FlashAttention3 Support
  ☆27Updated last year
• ziplab / QLLM
  [ICLR 2024] This is the official PyTorch implementation of "QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Mod…
  ☆30Updated last year
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆81Updated 2 years ago
• NVIDIA-NeMo / Emerging-Optimizers
  ☆115Updated 2 weeks ago
• srush / triton-autodiff
  Experiment of using Tangent to autodiff triton
  ☆81Updated last year
• epfml / pam
  ☆16Updated 2 years ago
• shikaiqiu / compute-better-spent
  ☆62Updated last year
• IST-DASLab / SparseFinetuning
  Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry
  ☆42Updated last year
• proger / nanokitchen
  Parallel Associative Scan for Language Models
  ☆18Updated 2 years ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆56Updated last year
• belindal / state-tracking
  Code and data for paper "(How) do Language Models Track State?"
  ☆21Updated 9 months ago
• chenjoya / dropit
  DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training (ICLR 2023)
  ☆32Updated 2 years ago
• aniquetahir / JORA
  JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)
  ☆36Updated last year
• shreyansh26 / Attention-Mask-Patterns
  Using FlexAttention to compute attention with different masking patterns
  ☆47Updated last year
• amirzandieh / HyperAttention
  Triton Implementation of HyperAttention Algorithm
  ☆48Updated 2 years ago
• VITA-Group / instant_soup
  [ICML2023] Instant Soup Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models. Ajay Jaiswal, Shiwei Liu, Ti…
  ☆11Updated 2 years ago
• hwang595 / Cuttlefish
  The implementation for MLSys 2023 paper: "Cuttlefish: Low-rank Model Training without All The Tuning"
  ☆45Updated 2 years ago
• RAIVNLab / MatFormer-OLMo
  Code repository for the public reproduction of the language modelling experiments on "MatFormer: Nested Transformer for Elastic Inference…
  ☆31Updated 2 years ago
• facebookresearch / Ternary_Binary_Transformer
  ACL 2023
  ☆39Updated 2 years ago
• deep-spin / adasplash
  AdaSplash: Adaptive Sparse Flash Attention (aka Flash Entmax Attention)
  ☆32Updated 3 months ago
• facebookresearch / adaptive_scheduling
  Experimental scripts for researching data adaptive learning rate scheduling.
  ☆22Updated 2 years ago
• IST-DASLab / MicroAdam
  This repository contains code for the MicroAdam paper.
  ☆21Updated last year
• PiotrNawrot / sparse-frontier
  The evaluation framework for training-free sparse attention in LLMs
  ☆108Updated 2 months ago
• shoaibahmed / llm_depth_pruning
  Official implementation of the paper: "A deeper look at depth pruning of LLMs"
  ☆15Updated last year
• epfml / schedules-and-scaling
  Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"
  ☆87Updated last year
• BBuf / flash-rwkv
  ☆32Updated last year
• Doraemonzzz / hgru2-pytorch
  ☆24Updated last year