intel / xFasterTransformer

Related projects ⓘ

Alternatives and complementary repositories for xFasterTransformer

• alibaba / rtp-llm
  RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
  ☆545Updated last month
• modelscope / dash-infer
  DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various hardware architectures, including …
  ☆137Updated 2 months ago
• pcg-mlp / KsanaLLM
  ☆290Updated this week
• intelligent-machine-learning / glake
  GLake: optimizing GPU memory management and IO transmission.
  ☆379Updated 3 months ago
• OpenPPL / ppl.llm.serving
  ☆123Updated 2 weeks ago
• hahnyuan / LLM-Viewer
  Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline mod…
  ☆311Updated 2 months ago
• ninehills / llm-inference-benchmark
  LLM Inference benchmark
  ☆350Updated 3 months ago
• bytedance / ByteMLPerf
  AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and ver…
  ☆206Updated this week
• LLMServe / DistServe
  Disaggregated serving system for Large Language Models (LLMs).
  ☆362Updated 3 months ago
• microsoft / T-MAC
  Low-bit LLM inference on CPU with lookup table
  ☆583Updated this week
• FlagOpen / FlagScale
  FlagScale is a large model toolkit based on open-sourced projects.
  ☆171Updated this week
• efeslab / Nanoflow
  A throughput-oriented high-performance serving framework for LLMs
  ☆637Updated 2 months ago
• FlagOpen / FlagGems
  FlagGems is an operator library for large language models implemented in Triton Language.
  ☆343Updated this week
• microsoft / vidur
  A large-scale simulation framework for LLM inference
  ☆277Updated last month
• AlibabaPAI / llumnix
  Efficient and easy multi-instance LLM serving
  ☆216Updated this week
• OpenPPL / ppl.nn.llm
  ☆140Updated 6 months ago
• ModelTC / llmc
  [EMNLP 2024 Industry Track] This is the official PyTorch implementation of "LLMC: Benchmarking Large Language Model Quantization with a V…
  ☆322Updated this week
• microsoft / vattention
  Dynamic Memory Management for Serving LLMs without PagedAttention
  ☆238Updated last week
• intel / llm-on-ray
  Pretrain, finetune and serve LLMs on Intel platforms with Ray
  ☆103Updated last week
• mit-han-lab / qserve
  QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
  ☆446Updated last week
• microsoft / sarathi-serve
  A low-latency & high-throughput serving engine for LLMs
  ☆245Updated 2 months ago
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆236Updated 8 months ago
• bytedance / ByteTransformer
  optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052
  ☆457Updated 8 months ago
• feifeibear / LLMRoofline
  Compare different hardware platforms via the Roofline Model for LLM inference tasks.
  ☆75Updated 8 months ago
• IST-DASLab / marlin
  FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
  ☆624Updated 2 months ago
• OpenPPL / ppl.llm.kernel.cuda
  ☆138Updated 2 weeks ago
• flashinfer-ai / flashinfer
  FlashInfer: Kernel Library for LLM Serving
  ☆1,452Updated this week
• volcengine / veScale
  A PyTorch Native LLM Training Framework
  ☆665Updated 2 months ago
• FlagOpen / FlagPerf
  FlagPerf is an open-source software platform for benchmarking AI chips.
  ☆313Updated this week
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆708Updated this week