tensorchord / inference-benchmarkLinks
Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)
☆28Updated 2 years ago
Alternatives and similar repositories for inference-benchmark
Users that are interested in inference-benchmark are comparing it to the libraries listed below
Sorting:
- A collection of reproducible inference engine benchmarks☆33Updated 5 months ago
- Benchmark suite for LLMs from Fireworks.ai☆83Updated this week
- Some microbenchmarks and design docs before commencement☆12Updated 4 years ago
- Simple dependency injection framework for Python☆21Updated last year
- WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.☆18Updated 3 years ago
- Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.☆103Updated last year
- ☆31Updated 5 months ago
- DPO, but faster 🚀☆44Updated 9 months ago
- ☆121Updated last year
- Manages vllm-nccl dependency☆17Updated last year
- Blazing fast data loading with HuggingFace Dataset and Ray Data☆16Updated last year
- Make triton easier☆47Updated last year
- vLLM adapter for a TGIS-compatible gRPC server.☆41Updated this week
- A memory efficient DLRM training solution using ColossalAI☆106Updated 2 years ago
- Sentence Embedding as a Service☆15Updated 3 months ago
- ☆56Updated 10 months ago
- Open sourced backend for Martian's LLM Inference Provider Leaderboard☆19Updated last year
- Repository for CPU Kernel Generation for LLM Inference☆26Updated 2 years ago
- Simple high-throughput inference library☆139Updated 4 months ago
- IBM development fork of https://github.com/huggingface/text-generation-inference☆61Updated 2 weeks ago
- ☆21Updated 7 months ago
- Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…☆56Updated last week
- The backend behind the LLM-Perf Leaderboard☆10Updated last year
- Repository for Sparse Finetuning of LLMs via modified version of the MosaicML llmfoundry☆42Updated last year
- experiments with inference on llama☆104Updated last year
- ☆15Updated 3 weeks ago
- A Python wrapper around HuggingFace's TGI (text-generation-inference) and TEI (text-embedding-inference) servers.☆33Updated 2 weeks ago
- LLM Serving Performance Evaluation Harness☆79Updated 7 months ago
- A place to store reusable transformer components of my own creation or found on the interwebs☆60Updated last week
- [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆128Updated 10 months ago