Yard1 / Ray-DeepSpeed-Inference

Related projects ⓘ

Alternatives and complementary repositories for Ray-DeepSpeed-Inference

• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆54Updated 7 months ago
• GAIR-NLP / Entropy-ABF
  Official implementation for 'Extending LLMs’ Context Window with 100 Samples'
  ☆74Updated 10 months ago
• neuralmagic / AutoFP8
  ☆158Updated last month
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆58Updated 2 weeks ago
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆130Updated 4 months ago
• CodeGeeX / codegeex-fastertransformer
  fastertransformer for codegeex model
  ☆64Updated last year
• alibaba / ChatLearn
  A flexible and efficient training framework for large-scale alignment tasks
  ☆209Updated this week
• MoFHeka / LLaMA-Megatron
  A LLaMA1/LLaMA12 Megatron implement.
  ☆27Updated 11 months ago
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆236Updated 8 months ago
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆96Updated 8 months ago
• shreyansh26 / Speculative-Sampling
  Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind
  ☆82Updated 8 months ago
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆35Updated 10 months ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆126Updated 5 months ago
• nyunAI / PruneGPT
  ☆53Updated 5 months ago
• Digitous / LLM-SLERP-Merge
  Spherical Merge Pytorch/HF format Language Models with minimal feature loss.
  ☆112Updated last year
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆76Updated last month
• AIR-Bench / AIR-Bench
  AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
  ☆106Updated last month
• bentoml / llm-bench
  ☆35Updated this week
• golololologol / LLM-Distillery
  A pipeline for LLM knowledge distillation
  ☆78Updated 3 months ago
• deepseek-ai / ESFT
  Expert Specialized Fine-Tuning
  ☆145Updated last month
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆176Updated this week
• anyscale / llm-continuous-batching-benchmarks
  ☆111Updated 8 months ago
• IsaacRe / vllm-kvcompress
  KV cache compression for high-throughput LLM inference
  ☆87Updated this week
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆124Updated 4 months ago
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆220Updated 3 weeks ago
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆42Updated 6 months ago
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆285Updated last month
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆177Updated 3 months ago
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆89Updated last year
• GAIR-NLP / ReAlign
  Reformatted Alignment
  ☆112Updated last month