runpod-workers / worker-sglang
SGLang is fast serving framework for large language models and vision language models.
☆10Updated this week
Related projects ⓘ
Alternatives and complementary repositories for worker-sglang
- Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding☆70Updated last week
- ☆53Updated 5 months ago
- Data preparation code for CrystalCoder 7B LLM☆42Updated 6 months ago
- Google TPU optimizations for transformers models☆74Updated last week
- QuIP quantization☆46Updated 7 months ago
- Pre-training code for CrystalCoder 7B LLM☆53Updated 6 months ago
- Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.☆172Updated 3 months ago
- A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.☆73Updated 3 weeks ago
- KV cache compression for high-throughput LLM inference☆82Updated last week
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters☆104Updated last month
- ☆73Updated 10 months ago
- The official repo for "LLoCo: Learning Long Contexts Offline"☆110Updated 4 months ago
- Data preparation code for Amber 7B LLM☆82Updated 6 months ago
- ☆92Updated last month
- Experiments on speculative sampling with Llama models☆117Updated last year
- ☆49Updated 7 months ago
- Layer-Condensed KV cache w/ 10 times larger batch size, fewer params and less computation. Dramatic speed up with better task performance…☆137Updated this week
- ☆116Updated last week
- Repo hosting codes and materials related to speeding LLMs' inference using token merging.☆29Updated 6 months ago
- ☆95Updated last month
- ☆43Updated 3 months ago
- ☆121Updated 9 months ago
- an implementation of Self-Extend, to expand the context window via grouped attention☆118Updated 10 months ago
- Spherical Merge Pytorch/HF format Language Models with minimal feature loss.☆111Updated last year
- An efficent implementation of the method proposed in "The Era of 1-bit LLMs"☆154Updated 3 weeks ago
- Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks☆31Updated 5 months ago
- The Benefits of a Concise Chain of Thought on Problem Solving in Large Language Models☆20Updated 9 months ago
- ☆96Updated last month
- ☆21Updated 5 months ago
- Repo for "LoLCATs: On Low-Rank Linearizing of Large Language Models"☆173Updated 3 weeks ago