Alternatives and similar repositories for Ray-DeepSpeed-Inference

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆131Updated last year
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆75Updated last year
• DachengLi1 / LongChat
  Official repository for LongChat and LongEval
  ☆531Updated last year
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆249Updated last year
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆152Updated last year
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆251Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆83Updated 2 weeks ago
• liyucheng09 / Selective_Context
  Compress your input to ChatGPT or other LLMs, to let them process 2x more content and save 40% memory and GPU time.
  ☆398Updated last year
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆97Updated last year
• bentoml / llm-bench
  ☆56Updated 11 months ago
• SomeoneKong / llm_long_context_bench202405
  ☆29Updated last year
• nickrosh / evol-teacher
  Open Source WizardCoder Dataset
  ☆161Updated 2 years ago
• LLM-inference-router / vllm-router
  vLLM Router
  ☆49Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆216Updated last year
• neuralmagic / AutoFP8
  ☆205Updated 5 months ago
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆233Updated 10 months ago
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆353Updated last year
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆210Updated last month
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆53Updated 2 years ago
• FranxYao / Long-Context-Data-Engineering
  Implementation of paper Data Engineering for Scaling Language Models to 128K Context
  ☆477Updated last year
• Re-Align / URIAL
  ☆313Updated last year
• OpenLMLab / ChatZoo
  Light local website for displaying performances from different chat models.
  ☆87Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆137Updated last year
• OpenLMLab / LEval
  [ACL'24 Outstanding] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
  ☆390Updated last year
• nick7nlp / Counting-Stars
  Counting-Stars (★)
  ☆83Updated 5 months ago
• AIR-Bench / AIR-Bench
  [ACL 2025] AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
  ☆160Updated 3 weeks ago
• OpenBMB / Eurus
  ☆320Updated last year
• nlpxucan / evol-instruct
  ☆274Updated 2 years ago
• shreyansh26 / Speculative-Sampling
  Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind
  ☆105Updated last year
• ninehills / llm-inference-benchmark
  LLM Inference benchmark
  ☆428Updated last year