Alternatives and similar repositories for Ray-DeepSpeed-Inference:

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆70Updated 2 months ago
• bentoml / llm-bench
  ☆50Updated 5 months ago
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆243Updated last year
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆95Updated last year
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆131Updated 10 months ago
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆54Updated last year
• nyunAI / PruneGPT
  ☆53Updated 11 months ago
• etalab-ia / albert-models
  Deployment a light and full OpenAI API for production with vLLM to support /v1/embeddings with all embeddings models.
  ☆42Updated 9 months ago
• cuplv / text-to-sql-wizardcoder
  Leveraging large language models for text-to-SQL synthesis, this project fine-tunes WizardLM/WizardCoder-15B-V1.0 with QLoRA on a custom …
  ☆43Updated last year
• CodeGeeX / codegeex-fastertransformer
  fastertransformer for codegeex model
  ☆63Updated last year
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆79Updated last year
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆50Updated 5 months ago
• llm-factory / imitater
  Imitate OpenAI with Local Models
  ☆88Updated 8 months ago
• triton-inference-server / vllm_backend
  ☆250Updated last week
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆125Updated last year
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆132Updated 10 months ago
• LLM-inference-router / vllm-router
  vLLM Router
  ☆27Updated last year
• GAIR-NLP / Entropy-ABF
  Official implementation for 'Extending LLMs’ Context Window with 100 Samples'
  ☆77Updated last year
• neuralmagic / AutoFP8
  ☆186Updated 7 months ago
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆41Updated last year
• SomeoneKong / llm_long_context_bench202405
  ☆29Updated 8 months ago
• OpenLMLab / LEval
  [ACL'24 Outstanding] Data and code for L-Eval, a comprehensive long context language models evaluation benchmark
  ☆376Updated 9 months ago
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆44Updated 11 months ago
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆205Updated 9 months ago
• OpenBMB / InfiniteBench
  Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718
  ☆323Updated 7 months ago
• GAIR-NLP / ReAlign
  Reformatted Alignment
  ☆115Updated 7 months ago
• ExpressAI / AI-Gaokao
  Gaokao Benchmark for AI
  ☆108Updated 2 years ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 10 months ago
• IEIT-Yuan / Yuan2.0-M32
  Mixture-of-Experts (MoE) Language Model
  ☆186Updated 7 months ago