alexisrozhkov / dilated-self-attention

Related projects ⓘ

Alternatives and complementary repositories for dilated-self-attention

• fkodom / dilated-attention-pytorch
  (Unofficial) Implementation of dilated attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens" (https://arxiv.org/abs/2307…
  ☆51Updated last year
• imoneoi / multipack_sampler
  Multipack distributed sampler for fast padding-free training of LLMs
  ☆178Updated 3 months ago
• flawedmatrix / mamba-ssm
  Implementation of mamba with rust
  ☆73Updated 8 months ago
• kjslag / spacebyte
  A byte-level decoder architecture that matches the performance of tokenized Transformers.
  ☆59Updated 7 months ago
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆89Updated last year
• booydar / LM-RMT
  Recurrent Memory Transformer
  ☆147Updated last year
• foundation-model-stack / fms-fsdp
  🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash…
  ☆194Updated this week
• llm-random / llm-random
  ☆176Updated this week
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆117Updated last month
• arcee-ai / PruneMe
  Automated Identification of Redundant Layer Blocks for Pruning in Large Language Models
  ☆196Updated 7 months ago
• jeffreysijuntan / lloco
  The official repo for "LLoCo: Learning Long Contexts Offline"
  ☆113Updated 5 months ago
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆90Updated 8 months ago
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆130Updated 2 months ago
• melisa-writer / short-transformers
  Prune transformer layers
  ☆65Updated 5 months ago
• EleutherAI / nanoGPT-mup
  The simplest, fastest repository for training/finetuning medium-sized GPTs.
  ☆84Updated this week
• catie-aq / flashT5
  A fast implementation of T5/UL2 in PyTorch using Flash Attention
  ☆71Updated last month
• Knowledgator / TurboT5
  Truly flash T5 realization!
  ☆54Updated 6 months ago
• HazyResearch / lolcats
  Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"
  ☆183Updated last month
• daniel-furman / sft-demos
  Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
  ☆64Updated last month
• microsoft / mutransformers
  some common Huggingface transformers in maximal update parametrization (µP)
  ☆77Updated 2 years ago
• EricLBuehler / xlora
  X-LoRA: Mixture of LoRA Experts
  ☆178Updated 3 months ago
• RobertCsordas / moe_attention
  Official repository for the paper "SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention"
  ☆93Updated last month
• mgmalek / efficient_cross_entropy
  ☆77Updated 5 months ago
• GreenBitAI / green-bit-llm
  A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
  ☆74Updated last month
• booydar / babilong
  BABILong is a benchmark for LLM evaluation using the needle-in-a-haystack approach.
  ☆157Updated this week
• qwopqwop200 / gptqlora
  GPTQLoRA: Efficient Finetuning of Quantized LLMs with GPTQ
  ☆97Updated last year
• srush / do-we-need-attention
  ☆161Updated last year
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆105Updated 5 months ago
• HazyResearch / zoology
  Understand and test language model architectures on synthetic tasks.
  ☆163Updated 6 months ago
• wdlctc / mini-s
  ☆35Updated 3 weeks ago