Alternatives and similar repositories for Ray-DeepSpeed-Inference

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆132Updated last year
• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆72Updated last year
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆248Updated last year
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆229Updated 9 months ago
• DachengLi1 / LongChat
  Official repository for LongChat and LongEval
  ☆533Updated last year
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆215Updated last year
• nickrosh / evol-teacher
  Open Source WizardCoder Dataset
  ☆161Updated 2 years ago
• bentoml / llm-bench
  ☆56Updated 10 months ago
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆149Updated last year
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆83Updated this week
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆467Updated last week
• liyucheng09 / Selective_Context
  Compress your input to ChatGPT or other LLMs, to let them process 2x more content and save 40% memory and GPU time.
  ☆397Updated last year
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆249Updated 11 months ago
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆97Updated last year
• myscale / Retrieval-QA-Benchmark
  Benchmark baseline for retrieval qa applications
  ☆116Updated last year
• triton-inference-server / vllm_backend
  ☆298Updated this week
• neuralmagic / nm-vllm
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆266Updated 11 months ago
• NetEase-FuXi / EETQ
  Easy and Efficient Quantization for Transformers
  ☆203Updated 3 months ago
• sabetAI / BLoRA
  batched loras
  ☆346Updated 2 years ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆224Updated last year
• FasterDecoding / REST
  REST: Retrieval-Based Speculative Decoding, NAACL 2024
  ☆209Updated 3 weeks ago
• ninehills / llm-inference-benchmark
  LLM Inference benchmark
  ☆426Updated last year
• FranxYao / Long-Context-Data-Engineering
  Implementation of paper Data Engineering for Scaling Language Models to 128K Context
  ☆473Updated last year
• anyscale / llm-continuous-batching-benchmarks
  ☆121Updated last year
• neuralmagic / AutoFP8
  ☆199Updated 4 months ago
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆56Updated 10 months ago
• microsoft / batch-inference
  Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
  ☆103Updated last year
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆53Updated 2 years ago
• ProjectD-AI / LLaMA-Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆69Updated 2 years ago
• nlpxucan / evol-instruct
  ☆275Updated 2 years ago