Alternatives and similar repositories for Ray-DeepSpeed-Inference

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆68Updated last year
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆131Updated last year
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆243Updated last year
• jshuadvd / LongRoPE
  Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
  ☆147Updated 11 months ago
• DachengLi1 / LongChat
  Official repository for LongChat and LongEval
  ☆523Updated last year
• lapp0 / lm-inference-engines
  Comparison of Language Model Inference Engines
  ☆219Updated 7 months ago
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆95Updated last year
• liyucheng09 / Selective_Context
  Compress your input to ChatGPT or other LLMs, to let them process 2x more content and save 40% memory and GPU time.
  ☆387Updated last year
• etalab-ia / albert-models
  Deployment a light and full OpenAI API for production with vLLM to support /v1/embeddings with all embeddings models.
  ☆42Updated last year
• CrazyBoyM / CodeLLaMA-chat
  CodeLLaMA 中文版 - 代码生成助手，huggingface累积下载2w+次
  ☆45Updated last year
• nlpxucan / evol-instruct
  ☆270Updated 2 years ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆76Updated this week
• npuichigo / openai_trtllm
  OpenAI compatible API for TensorRT LLM triton backend
  ☆209Updated 11 months ago
• HuangLK / transpeeder
  train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
  ☆223Updated last year
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆128Updated 2 years ago
• bentoml / llm-bench
  ☆55Updated 7 months ago
• CodeGeeX / codegeex-fastertransformer
  fastertransformer for codegeex model
  ☆63Updated 2 years ago
• microsoft / batch-inference
  Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
  ☆99Updated 11 months ago
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆54Updated 8 months ago
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆53Updated 2 years ago
• myscale / Retrieval-QA-Benchmark
  Benchmark baseline for retrieval qa applications
  ☆115Updated last year
• OpenMatch / NeuScraper
  [ACL 2024] This is the code repo for our ACL’24 paper "Cleaner Pretraining Corpus Curation with Neural Web Scraping".
  ☆226Updated 10 months ago
• nickrosh / evol-teacher
  Open Source WizardCoder Dataset
  ☆159Updated 2 years ago
• LLM-inference-router / vllm-router
  vLLM Router
  ☆31Updated last year
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆244Updated 8 months ago
• shreyansh26 / Speculative-Sampling
  Implementation of Speculative Sampling as described in "Accelerating Large Language Model Decoding with Speculative Sampling" by Deepmind
  ☆98Updated last year
• sabetAI / BLoRA
  batched loras
  ☆344Updated last year
• lilbillybiscuit / chatbot-ui-toolllama
  An open source ChatGPT UI for ToolLlama
  ☆28Updated last year
• young-geng / koala_data_pipeline
  The data processing pipeline for the Koala chatbot language model
  ☆117Updated 2 years ago
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆42Updated last year