godaai / llm-inference

Related projects ⓘ

Alternatives and complementary repositories for llm-inference

• neuralmagic / AutoFP8
  ☆156Updated last month
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆101Updated last week
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆74Updated 7 months ago
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆236Updated 7 months ago
• sgl-project / sgl-learning-materials
  Materials for learning SGLang
  ☆78Updated this week
• anyscale / llm-continuous-batching-benchmarks
  ☆109Updated 7 months ago
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆178Updated 3 months ago
• bytedance / ByteMLPerf
  AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…
  ☆200Updated 3 weeks ago
• wejoncy / QLLM
  A general 2-8 bits quantization toolbox with GPTQ/AWQ/HQQ, and export to onnx/onnx-runtime easily.
  ☆148Updated last month
• vllm-project / flash-attention
  Fast and memory-efficient exact attention
  ☆28Updated 2 weeks ago
• OpenPPL / ppl.nn.llm
  ☆140Updated 6 months ago
• casper-hansen / AutoAWQ_kernels
  ☆52Updated 2 months ago
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆22Updated this week
• microsoft / DeepSpeed-Kernels
  ☆55Updated 5 months ago
• ModelTC / awesome-lm-system
  Summary of system papers/frameworks/codes/tools on training or serving large model
  ☆56Updated 10 months ago
• OpenPPL / ppl.llm.serving
  ☆123Updated this week
• AlibabaPAI / FLASHNN
  ☆79Updated 2 months ago
• Ascend / AscendSpeed
  ☆74Updated 10 months ago
• tgale96 / grouped_gemm
  PyTorch bindings for CUTLASS grouped GEMM.
  ☆51Updated last week
• tlc-pack / libflash_attn
  Standalone Flash Attention v2 kernel without libtorch dependency
  ☆98Updated 2 months ago
• madsys-dev / deepseekv2-profile
  ☆63Updated 3 months ago
• nbasyl / LLM-FP4
  The official implementation of the EMNLP 2023 paper LLM-FP4
  ☆166Updated 10 months ago
• FlagOpen / FlagScale
  FlagScale is a large model toolkit based on open-sourced projects.
  ☆167Updated this week
• mlcommons / logging
  MLPerf™ logging library
  ☆30Updated last week
• triton-inference-server / vllm_backend
  ☆189Updated this week
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆127Updated 4 months ago
• HabanaAI / vllm-fork
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆41Updated this week
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆231Updated last month
• AlibabaPAI / llumnix
  Efficient and easy multi-instance LLM serving
  ☆194Updated this week
• Oneflow-Inc / serving
  OneFlow Serving
  ☆20Updated 9 months ago