Alternatives and similar repositories for dilated-self-attention

Users that are interested in dilated-self-attention are comparing it to the libraries listed below

• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆97Updated 8 months ago
• fkodom / dilated-attention-pytorch
  (Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens" (https://arxiv.org/abs/2307…
  ☆51Updated last year
• imoneoi / multipack_sampler
  Multipack distributed sampler for fast padding-free training of LLMs
  ☆190Updated 9 months ago
• Digitous / LLM-SLERP-Merge
  Spherical Merge Pytorch/HF format Language Models with minimal feature loss.
  ☆124Updated last year
• qwopqwop200 / gptqlora
  GPTQLoRA: Efficient Finetuning of Quantized LLMs with GPTQ
  ☆103Updated 2 years ago
• HanGuo97 / lq-lora
  ☆125Updated last year
• aju22 / LLaMA2
  This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…
  ☆67Updated last year
• HazyResearch / zoology
  Understand and test language model architectures on synthetic tasks.
  ☆197Updated 3 months ago
• EleutherAI / improved-t5
  Experiments for efforts to train a new and improved t5
  ☆77Updated last year
• microsoft / mutransformers
  some common Huggingface transformers in maximal update parametrization (µP)
  ☆80Updated 3 years ago
• hadasah / btm
  ☆74Updated last year
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆126Updated last year
• schwartz-lab-NLP / TOVA
  Token Omission Via Attention
  ☆126Updated 7 months ago
• OpenEvaByte / evabyte
  EvaByte: Efficient Byte-level Language Models at Scale
  ☆101Updated last month
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆95Updated last year
• SeunghyunSEO / optimized_hf_llama_class_for_training
  ☆47Updated 9 months ago
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆143Updated 8 months ago
• Oxen-AI / mamba-dive
  This is the code that went into our practical dive using mamba as information extraction
  ☆53Updated last year
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆161Updated 11 months ago
• arcee-ai / PruneMe
  Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models
  ☆238Updated last year
• Pleias / Various-Finetuning
  Set of scripts to finetune LLMs
  ☆37Updated last year
• BlinkDL / modded-nanogpt-rwkv
  RWKV-7: Surpassing GPT
  ☆88Updated 6 months ago
• kyleliang919 / Long-context-transformers
  Exploring finetuning public checkpoints on filter 8K sequences on Pile
  ☆114Updated 2 years ago
• llm-random / llm-random
  ☆190Updated this week
• FasterDecoding / BitDelta
  ☆197Updated 6 months ago
• euclaise / supertrainer2000
  ☆49Updated last year
• wdlctc / mini-s
  ☆51Updated 7 months ago
• mgmalek / efficient_cross_entropy
  ☆108Updated last year
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆56Updated 3 weeks ago
• IST-DASLab / peft-rosa
  A fork of the PEFT library, supporting Robust Adaptation (RoSA)
  ☆14Updated 9 months ago