sgl-project / tensorrt-demo
TensorRT LLM Benchmark Configuration
☆12Updated 5 months ago
Alternatives and similar repositories for tensorrt-demo:
Users that are interested in tensorrt-demo are comparing it to the libraries listed below
- Open deep learning compiler stack for cpu, gpu and specialized accelerators☆17Updated 2 weeks ago
- GPTQ inference TVM kernel☆38Updated 8 months ago
- ☆36Updated this week
- Framework to reduce autotune overhead to zero for well known deployments.☆57Updated last month
- Benchmark tests supporting the TiledCUDA library.☆12Updated last month
- 📚[WIP] FFPA: Yet antother Faster Flash Prefill Attention with O(1)⚡️GPU SRAM complexity for headdim > 256, 1.8x~3x↑🎉faster vs SDPA EA.☆49Updated this week
- Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.☆27Updated 2 months ago
- Quantized Attention on GPU☆34Updated last month
- APPy (Annotated Parallelism for Python) enables users to annotate loops and tensor expressions in Python with compiler directives akin to…☆21Updated 2 weeks ago
- Tritonbench is a collection of PyTorch custom operators with example inputs to measure their performance.☆75Updated this week
- Odysseus: Playground of LLM Sequence Parallelism☆64Updated 7 months ago
- Transformers components but in Triton☆29Updated 2 months ago
- ☆57Updated 7 months ago
- ☆19Updated 3 months ago
- FlexAttention w/ FlashAttention3 Support☆27Updated 3 months ago
- A minimal implementation of vllm.☆32Updated 5 months ago
- FlexFlow Serve: Low-Latency, High-Performance LLM Serving☆16Updated this week
- ⚡️Write HGEMM from scratch using Tensor Cores with WMMA, MMA and CuTe API, achieve peak⚡️ performance☆43Updated this week
- ☆21Updated last week
- ☆56Updated 3 months ago
- ☆22Updated 3 weeks ago
- 使用 CUDA C++ 实现的 llama 模型推理框架☆43Updated 2 months ago
- A CUDA kernel for NHWC GroupNorm for PyTorch☆16Updated 2 months ago
- ☆22Updated last month
- Standalone Flash Attention v2 kernel without libtorch dependency☆99Updated 4 months ago
- Multiple GEMM operators are constructed with cutlass to support LLM inference.☆16Updated 3 months ago
- ☆78Updated 4 months ago
- Repository for CPU Kernel Generation for LLM Inference☆25Updated last year
- A standalone GEMM kernel for fp16 activation and quantized weight, extracted from FasterTransformer☆87Updated 10 months ago