Alternatives and similar repositories for Ray-DeepSpeed-Inference:

Users that are interested in Ray-DeepSpeed-Inference are comparing it to the libraries listed below

• asprenger / ray_vllm_inference
  A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
  ☆65Updated 11 months ago
• llm-factory / imitater
  Imitate OpenAI with Local Models
  ☆88Updated 6 months ago
• fw-ai / benchmark
  Benchmark suite for LLMs from Fireworks.ai
  ☆69Updated last month
• GAIR-NLP / Entropy-ABF
  Official implementation for 'Extending LLMs’ Context Window with 100 Samples'
  ☆75Updated last year
• ninehills / langeval
  Evaluation for AI apps and agent
  ☆36Updated last year
• chu-tianxiang / vllm-gptq
  A high-throughput and memory-efficient inference and serving engine for LLMs
  ☆130Updated 8 months ago
• OpenBMB / UltraEval
  [ACL 2024 Demo] Official GitHub repo for UltraEval: An open source framework for evaluating foundation models.
  ☆234Updated 4 months ago
• cuplv / text-to-sql-wizardcoder
  Leveraging large language models for text-to-SQL synthesis, this project fine-tunes WizardLM/WizardCoder-15B-V1.0 with QLoRA on a custom …
  ☆43Updated last year
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆36Updated last year
• thu-coai / SPaR
  ☆42Updated 3 months ago
• golololologol / LLM-Distillery
  A pipeline for LLM knowledge distillation
  ☆98Updated last month
• LowinLi / transformers-stream-generator
  This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…
  ☆95Updated last year
• inferflow / inferflow
  Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
  ☆238Updated last year
• LLM360 / crystalcoder-data-prep
  Data preparation code for CrystalCoder 7B LLM
  ☆44Updated 10 months ago
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆125Updated last year
• nyunAI / PruneGPT
  ☆53Updated 9 months ago
• OpenMatch / NeuScraper
  [ACL 2024] This is the code repo for our ACL’24 paper "Cleaner Pretraining Corpus Curation with Neural Web Scraping".
  ☆224Updated 6 months ago
• zhaochenyang20 / ModelServer
  Efficient, Flexible, and Highly Fault-Tolerant Model Service Management Based on SGLang
  ☆42Updated 4 months ago
• liuqi6777 / pe_rank
  Leveraging passage embeddings for efficient listwise reranking with large language models.
  ☆38Updated 3 months ago
• AIR-Bench / AIR-Bench
  AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
  ☆132Updated 3 months ago
• SkyworkAI / Skywork-MoE
  Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models
  ☆129Updated 9 months ago
• OpenLMLab / ChatZoo
  Light local website for displaying performances from different chat models.
  ☆85Updated last year
• FlagOpen / Infinity-Instruct
  ☆45Updated 9 months ago
• microsoft / batch-inference
  Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
  ☆94Updated 7 months ago
• SivilTaram / code-html-to-markdown
  A lightweight script for processing HTML page to markdown format with support for code blocks
  ☆79Updated 11 months ago
• HITsz-TMG / KaLM-Embedding
  Code for KaLM-Embedding models
  ☆74Updated this week
• THU-KEG / ChatLog
  ⏳ ChatLog: Recording and Analysing ChatGPT Across Time
  ☆97Updated 9 months ago
• SomeoneKong / llm_long_context_bench202405
  ☆29Updated 6 months ago
• multimodal-art-projection / Megatron-LM-NEO
  ☆37Updated 10 months ago
• etalab-ia / albert-models
  Deployment a light and full OpenAI API for production with vLLM to support /v1/embeddings with all embeddings models.
  ☆41Updated 8 months ago