Harry-Chen / InfMoE
Inference framework for MoE layers based on TensorRT with Python binding
☆41Updated 3 years ago
Alternatives and similar repositories for InfMoE:
Users that are interested in InfMoE are comparing it to the libraries listed below
- Odysseus: Playground of LLM Sequence Parallelism☆64Updated 7 months ago
- ☆59Updated last week
- Distributed DataLoader For Pytorch Based On Ray☆24Updated 3 years ago
- PSTensor provides a way to hack the memory management of tensors in TensorFlow and PyTorch by defining your own C++ Tensor Class.☆10Updated 3 years ago
- GPTQ inference TVM kernel☆38Updated 9 months ago
- ☆26Updated 3 years ago
- ☆22Updated last year
- Summary of system papers/frameworks/codes/tools on training or serving large model☆56Updated last year
- ☆81Updated 5 months ago
- TileFusion is a highly efficient kernel template library designed to elevate the level of abstraction in CUDA C for processing tiles.☆53Updated this week
- Framework to reduce autotune overhead to zero for well known deployments.☆61Updated 2 weeks ago
- A Python library transfers PyTorch tensors between CPU and NVMe☆103Updated 2 months ago
- ☆70Updated 3 years ago
- ☆99Updated 5 months ago
- pytorch-profiler☆50Updated last year
- Fairring (FAIR + Herring) is a plug-in for PyTorch that provides a process group for distributed training that outperforms NCCL at large …☆64Updated 2 years ago
- Standalone Flash Attention v2 kernel without libtorch dependency☆103Updated 5 months ago
- ☆75Updated 2 years ago
- PyTorch bindings for CUTLASS grouped GEMM.☆64Updated 3 months ago
- High-speed GEMV kernels, at most 2.7x speedup compared to pytorch baseline.☆97Updated 7 months ago
- Fast and memory-efficient exact attention☆44Updated this week
- PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections☆117Updated 2 years ago
- ATC23 AE☆45Updated last year
- A plug-in of Microsoft DeepSpeed to fix the bug of DeepSpeed pipeline☆26Updated 3 years ago
- Automated Parallelization System and Infrastructure for Multiple Ecosystems☆78Updated 2 months ago
- Boosting 4-bit inference kernels with 2:4 Sparsity☆64Updated 5 months ago
- Tritonbench is a collection of PyTorch custom operators with example inputs to measure their performance.☆86Updated this week
- Official repository for LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers☆205Updated 5 months ago
- A standalone GEMM kernel for fp16 activation and quantized weight, extracted from FasterTransformer☆88Updated 11 months ago