High-Throughput Batch Inference
☆18Oct 16, 2026Updated this week
Alternatives and similar repositories for batchgen
Users that are interested in batchgen are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- LLM-powered Python☆16Apr 8, 2026Updated 6 months ago
- A lightweight post-training framework for LLMs and VLMs. 51 algorithms, 38 verified models. Scales with DeepSpeed, vLLM, and Ray.☆20Aug 4, 2026Updated 2 months ago
- BGEMM-CUDA is a CUDA-based low-bit GEMM kernel library for efficient neural network inference. It implements optimized binary and ternary…☆21Aug 30, 2024Updated 2 years ago
- [HPCA 2026] A GPU-optimized system for efficient long-context LLMs decoding with low-bit KV cache.☆102May 14, 2026Updated 5 months ago
- ☆10Mar 14, 2020Updated 6 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- ☆31Feb 3, 2026Updated 8 months ago
- PyTorch library for cost-effective, fast and easy serving of MoE models.☆373Updated this week
- Examples and experiments of GBDT for multiple outputs.☆13Jan 20, 2025Updated last year
- [ICDCS 2023] Evaluation and Optimization of Gradient Compression for Distributed Deep Learning☆10Apr 28, 2023Updated 3 years ago
- Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM, llama.cpp, OpenClaw, RAG, and Agen…☆141Updated this week
- HKUST(GZ) Marpit Template for Presentation☆10Nov 21, 2022Updated 3 years ago
- ☆32May 28, 2024Updated 2 years ago
- ☆52Dec 26, 2024Updated last year
- 《Machine Learning Systems: Design and Implementation》- English Version☆41Jan 27, 2025Updated last year
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Quantized Attention on GPU☆45Nov 22, 2024Updated last year
- [AFK] Hardware router in Chisel (THU Network Joint Lab 2020)☆14Oct 8, 2020Updated 6 years ago
- Code for MLSys 2024 Paper "SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models"☆23Apr 13, 2024Updated 2 years ago
- More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression（EMNLP 2025）☆12Jan 15, 2025Updated last year
- vllm based inference engine for looped transformer☆192Updated this week
- Compiler for Dynamic Neural Networks☆45Nov 13, 2023Updated 2 years ago
- Reading seminar in Harvard Cloud Networking and Systems Group☆16Aug 29, 2022Updated 4 years ago
- SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs☆70Mar 25, 2025Updated last year
- Repository to reproduce SketchBoost paper experiments☆23Dec 20, 2022Updated 3 years ago
- Virtual machines for every use case on DigitalOcean • AdGet dependable uptime with 99.99% SLA, simple security tools, and predictable monthly pricing with DigitalOcean's virtual machines, called Droplets.
- ☆17May 10, 2024Updated 2 years ago
- Benchmark tests supporting the TiledCUDA library.☆19Nov 19, 2024Updated last year
- [Archived] For the latest updates and community contribution, please visit: https://github.com/Ascend/TransferQueue or https://gitcode.co…☆15Aug 14, 2026Updated 2 months ago
- ☆10May 14, 2023Updated 3 years ago
- SC'25 UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling☆16Aug 14, 2025Updated last year
- AI model training on heterogeneous, geo-distributed resources☆46Nov 24, 2025Updated 10 months ago
- Official repository for the paper DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines☆19Dec 8, 2023Updated 2 years ago
- ☆17Mar 24, 2026Updated 6 months ago
- [ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs☆135Oct 13, 2026Updated last week
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion (NeurIPS 2024 Spotlight)☆15Mar 31, 2025Updated last year
- APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation. A system-level optimization for scalable LLM tra…☆64Oct 11, 2025Updated last year
- DeDe (OSDI '25): an optimization framework for large-scale resource allocation☆15May 18, 2026Updated 5 months ago
- An extention of TVMScript to write simple and high performance GPU kernels with tensorcore.☆52Jul 23, 2024Updated 2 years ago
- Claude Code skill: Generate file-by-file code tutorial websites for any repository with parallel agent teams☆28Mar 13, 2026Updated 7 months ago
- Summary of the Specs of Commonly Used GPUs for Training and Inference of LLM☆78Aug 12, 2025Updated last year
- Tutorials of Extending and importing TVM with CMAKE Include dependency.☆16Oct 11, 2024Updated 2 years ago