hpcaitech / CachedEmbedding

Related projects: ⓘ

• anyscale / llm-continuous-batching-benchmarks
  ☆108Updated 6 months ago
• hpcaitech / TensorNVMe
  A Python library transfers PyTorch tensors between CPU and NVMe
  ☆92Updated last year
• Harry-Chen / InfMoE
  Inference framework for MoE layers based on TensorRT with Python binding
  ☆41Updated 3 years ago
• stanford-futuredata / stk
  ☆83Updated 3 weeks ago
• microsoft / DeepSpeed-Kernels
  ☆50Updated 3 months ago
• alibaba / ChatLearn
  ☆105Updated last week
• yale-sys / prompt-cache
  Modular and structured prompt caching for low-latency LLM inference
  ☆43Updated 4 months ago
• Ascend / AscendSpeed
  ☆70Updated 9 months ago
• zms1999 / SmartMoE
  A MoE impl for PyTorch, [ATC'23] SmartMoE
  ☆56Updated last year
• LMCache / LMCache
  ☆32Updated last week
• RulinShao / LightSeq
  Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers
  ☆183Updated last month
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆71Updated 6 months ago
• feifeibear / long-context-attention
  USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
  ☆309Updated this week
• TorchMoE / MoE-Infinity
  PyTorch library for cost-effective, fast and easy serving of MoE models.
  ☆90Updated last month
• thu-pacman / SmartMoE-AE
  ATC23 AE
  ☆42Updated last year
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆50Updated 3 months ago
• hpcaitech / Titans
  A collection of models built with ColossalAI
  ☆32Updated last year
• ModelTC / awesome-lm-system
  Summary of system papers/frameworks/codes/tools on training or serving large model
  ☆56Updated 9 months ago
• usyd-fsalab / fp6_llm
  An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).
  ☆173Updated 3 months ago
• madsys-dev / deepseekv2-profile
  ☆60Updated last month
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆174Updated last week
• IST-DASLab / Sparse-Marlin
  Boosting 4-bit inference kernels with 2:4 Sparsity
  ☆47Updated 2 weeks ago
• volcengine / veGiantModel
  ☆205Updated last year
• fpgaminer / GPTQ-triton
  GPTQ inference Triton kernel
  ☆273Updated last year
• efeslab / Atom
  [MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
  ☆258Updated 2 months ago
• hpcaitech / Elixir
  Elixir: Train a Large Language Model on a Small GPU Cluster
  ☆12Updated last year
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆109Updated 11 months ago
• Hsword / Hetu
  A high-performance distributed deep learning system targeting large-scale and automated distributed training. If you have any interests, …
  ☆101Updated 9 months ago
• FMInference / H2O
  [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
  ☆360Updated last month
• fanshiqing / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆57Updated 2 months ago