Alternatives and similar repositories for Ray-DeepSpeed-Inference

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆67Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆76Updated 2 weeks ago
• nyunAI / PruneGPT
  ☆53Updated last year
• lilbillybiscuit / chatbot-ui-toolllama
  An open source ChatGPT UI for ToolLlama
  ☆28Updated last year
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆79Updated last year
• anyscale / llm-continuous-batching-benchmarks
  ☆119Updated last year
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆95Updated last year
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆131Updated 11 months ago
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆242Updated last year
• CodeGeeX / codegeex-fastertransformer
  fastertransformer for codegeex model
  ☆63Updated 2 years ago
• etalab-ia / albert-models
  Deployment a light and full OpenAI API for production with vLLM to support /v1/embeddings with all embeddings models.
  ☆42Updated 11 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated last year
• substratusai / vllm-docker
  ☆62Updated 2 months ago
• togethercomputer / Llama-2-7B-32K-Instruct
  ☆84Updated last year
• mlc-ai / llm-perf-bench
  ☆118Updated last year
• kemingy / vllm-env
  setup the env for vllm users
  ☆16Updated last year
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆128Updated 2 years ago
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆45Updated last year
• GAIR-NLP / Entropy-ABF
  Official implementation for 'Extending LLMs’ Context Window with 100 Samples'
  ☆78Updated last year
• bentoml / llm-bench
  ☆54Updated 7 months ago
• xlang-ai / batch-prompting
  [EMNLP 2023 Industry Track] A simple prompting approach that enables the LLMs to run inference in batches.
  ☆74Updated last year
• AIR-Bench / AIR-Bench
  AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
  ☆144Updated 6 months ago
• nickrosh / evol-teacher
  Open Source WizardCoder Dataset
  ☆158Updated last year
• jquesnelle / transformers-openai-api
  An OpenAI Completions API compatible server for NLP transformers models
  ☆65Updated last year
• Infini-AI-Lab / MagicDec
  [ICLR2025] Breaking Throughput-Latency Trade-off for Long Sequences with Speculative Decoding
  ☆116Updated 6 months ago
• jaymody / speculative-sampling
  Simple implementation of Speculative Sampling in NumPy for GPT-2.
  ☆95Updated last year
• microsoft / ToolTalk
  Evaluating tool-augmented LLMs in conversation settings
  ☆85Updated last year
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆42Updated last year
• cuplv / text-to-sql-wizardcoder
  Leveraging large language models for text-to-SQL synthesis, this project fine-tunes WizardLM/WizardCoder-15B-V1.0 with QLoRA on a custom …
  ☆44Updated last year
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆53Updated 7 months ago