Alternatives and similar repositories for Ray-DeepSpeed-Inference

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆132Updated last year
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆251Updated last year
• DachengLi1 / LongChat
  Official repository for LongChat and LongEval
  ☆534Updated last year
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆239Updated last year
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆78Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆220Updated last year
• ninehills / llm-inference-benchmark
  LLM Inference benchmark
  ☆433Updated last year
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆96Updated last year
• sabetAI / BLoRA
  batched loras
  ☆349Updated 2 years ago
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆215Updated 4 months ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆89Updated this week
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆255Updated last year
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆150Updated last year
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆204Updated last week
• nickrosh / evol-teacher
  Open Source WizardCoder Dataset
  ☆163Updated 2 years ago
• neuralmagic / AutoFP8
  ☆206Updated 9 months ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆224Updated 2 years ago
• SomeoneKong / llm_long_context_bench202405
  ☆29Updated last year
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆491Updated last month
• bentoml / llm-bench
  ☆56Updated last year
• OpenLMLab / ChatZoo
  Light local website for displaying performances from different chat models.
  ☆87Updated 2 years ago
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆370Updated last year
• anyscale / llm-continuous-batching-benchmarks
  ☆125Updated last year
• hpcaitech / SwiftInfer
  Efficient AI Inference & Serving
  ☆479Updated 2 years ago
• bigscience-workshop / data-preparation
  Code used for sourcing and cleaning the BigScience ROOTS corpus
  ☆318Updated 2 years ago
• void-main / fastertransformer_backend
  ☆22Updated 2 years ago
• llm-semantic-router / vllm-router
  vLLM Router
  ☆54Updated last year
• the-seeds / imitater
  Imitate OpenAI with Local Models
  ☆90Updated last year
• QwenLM / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆140Updated last year
• microsoft / batch-inference
  Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
  ☆106Updated last year