google-deepmind / language_modeling_is_compression

Related projects ⓘ

Alternatives and complementary repositories for language_modeling_is_compression

• hkust-nlp / llm-compression-intelligence
  Official github repo for the paper "Compression Represents Intelligence Linearly" [COLM 2024]
  ☆127Updated last month
• siyan-zhao / prepacking
  The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models"
  ☆56Updated 3 weeks ago
• ryoungj / ObsScaling
  [NeurIPS'24 Spotlight] Observational Scaling Laws
  ☆42Updated last month
• TIGER-AI-Lab / LongICLBench
  Code and Data for "Long-context LLMs Struggle with Long In-context Learning"
  ☆91Updated 4 months ago
• princeton-nlp / ProLong
  Homepage for ProLong (Princeton long-context language models) and paper "How to Train Long-Context Language Models (Effectively)"
  ☆111Updated last week
• nightdessert / Retrieval_Head
  open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality
  ☆156Updated 3 months ago
• epfml / dynamic-sparse-flash-attention
  ☆132Updated last year
• astramind-ai / Mixture-of-depths
  Unofficial implementation for the paper "Mixture-of-Depths: Dynamically allocating compute in transformer-based language models"
  ☆134Updated 4 months ago
• vwxyzjn / summarize_from_feedback_details
  ☆112Updated 3 months ago
• sail-sg / scaling-with-vocab
  [NeurIPS-2024] 📈 Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies https://arxiv.org/abs/2407.13623
  ☆67Updated last month
• RobertCsordas / moeut
  ☆61Updated 2 months ago
• jongwooko / distillm
  Official PyTorch implementation of DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024)
  ☆133Updated last month
• mlfoundations / scaling
  Language models scale reliably with over-training and on downstream tasks
  ☆94Updated 7 months ago
• whyNLP / LCKV
  Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…
  ☆137Updated this week
• Edward-Sun / gpt-accelera
  Simple and efficient pytorch-native transformer training and inference (batched)
  ☆61Updated 7 months ago
• UNITES-Lab / MC-SMoE
  [ICLR 2024 Spotlight] Code for the paper "Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy"
  ☆64Updated 5 months ago
• berlino / gated_linear_attention
  ☆97Updated 8 months ago
• locuslab / massive-activations
  Code accompanying the paper "Massive Activations in Large Language Models"
  ☆121Updated 8 months ago
• JacobPfau / fillerTokens
  ☆49Updated 6 months ago
• berlino / seq_icl
  ☆50Updated 5 months ago
• hkust-nlp / dart-math
  [NeurIPS'24] Official code for *🎯DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving*
  ☆75Updated last month
• McGill-NLP / VinePPO
  Code for the paper "VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment"
  ☆77Updated 2 weeks ago
• shawntan / stickbreaking-attention
  Stick-breaking attention
  ☆33Updated this week
• jzhang38 / LongMamba
  Some preliminary explorations of Mamba's context scaling.
  ☆190Updated 9 months ago
• gregorbachmann / Next-Token-Failures
  ☆65Updated 7 months ago
• wuhy68 / Parameter-Efficient-MoE
  Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  ☆129Updated last month
• allenai / WildBench
  Benchmarking LLMs with Challenging Tasks from Real Users
  ☆194Updated last week
• thunlp / MoEfication
  ☆107Updated 3 months ago
• roeehendel / icl_task_vectors
  ☆79Updated last year
• epfml / schedules-and-scaling
  ☆50Updated last week