Alternatives and similar repositories for Knowledge-Distillation

Users that are interested in Knowledge-Distillation are comparing it to the libraries listed below

• Montinger / Transformer-Workbench
  Playground for Transformers
  ☆52Updated last year
• rasbt / cvpr2023
  ☆132Updated last year
• knotgrass / attention
  several types of attention modules written in PyTorch for learning purposes
  ☆53Updated 8 months ago
• hkproj / pytorch-transformer-distributed
  Distributed training (multi-node) of a Transformer model
  ☆68Updated last year
• ariG23498 / quantized-diffusion-inference
  Notebook and Scripts that showcase running quantized diffusion models on consumer GPUs
  ☆38Updated 7 months ago
• mishra-18 / ML-Models
  ☆38Updated last month
• hkproj / quantization-notes
  Notes on quantization in neural networks
  ☆83Updated last year
• hkproj / pytorch-lora
  LORA: Low-Rank Adaptation of Large Language Models implemented using PyTorch
  ☆105Updated last year
• rasbt / dora-from-scratch
  LoRA and DoRA from Scratch Implementations
  ☆204Updated last year
• fangyuan-ksgk / Mini-LLaVA
  A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.
  ☆93Updated 5 months ago
• ThinamXx / cuda-mode
  Making of cuda kernel
  ☆16Updated last week
• tanaymeh / mamba-train
  A single repo with all scripts and utils to train / fine-tune the Mamba model with or without FIM
  ☆54Updated last year
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆26Updated 7 months ago
• pengzhangzhi / Awesome-Mamba
  Awesome list of papers that extend Mamba to various applications.
  ☆133Updated 2 months ago
• IST-DASLab / peft-rosa
  A fork of the PEFT library, supporting Robust Adaptation (RoSA)
  ☆14Updated 9 months ago
• rasbt / pytorch-memory-optim
  This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…
  ☆91Updated last year
• NielsRogge / awesome-huggingface
  Repository containing awesome resources regarding Hugging Face tooling.
  ☆47Updated last year
• ThinamXx / Meta-llama
  Complete implementation of Llama2 with/without KV cache & inference 🚀
  ☆46Updated last year
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 8 months ago
• kyegomez / Mixture-of-Depths
  Implementation of the paper: "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆94Updated this week
• ShaderManager / RetNet
  PyTorch implementation of Retentive Network: A Successor to Transformer for Large Language Models
  ☆14Updated last year
• lucidrains / deep-cross-attention
  Implementation of the proposed DeepCrossAttention by Heddes et al at Google research, in Pytorch
  ☆86Updated 3 months ago
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆85Updated last month
• kyegomez / MoE-Mamba
  Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Ze…
  ☆105Updated 2 months ago
• joey00072 / Multi-Head-Latent-Attention-MLA-
  working implimention of deepseek MLA
  ☆41Updated 4 months ago
• hkproj / mamba-notes
  Notes on the Mamba and the S4 model (Mamba: Linear-Time Sequence Modeling with Selective State Spaces)
  ☆168Updated last year
• kyegomez / TTL
  Pytorch Implementation of the paper: "Learning to (Learn at Test Time): RNNs with Expressive Hidden States"
  ☆24Updated this week
• kabir2505 / tiny-mixtral
  ☆39Updated last month
• jacobmarks / awesome-neurips-2023
  Conference schedule, top papers, and analysis of the data for NeurIPS 2023!
  ☆119Updated last year
• Weixin-Liang / Mixture-of-Mamba
  ☆43Updated 4 months ago