mosecorg / mosec

Related projects: ⓘ

• triton-inference-server / fastertransformer_backend
  ☆411Updated 10 months ago
• punica-ai / punica
  Serving multiple LoRA finetuned LLM as one
  ☆946Updated 4 months ago
• triton-inference-server / python_backend
  Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
  ☆523Updated this week
• microsoft / DeepSpeed-MII
  MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
  ☆1,843Updated last week
• triton-inference-server / pytriton
  PyTriton is a Flask/FastAPI-like interface that simplifies Triton's deployment in Python environments.
  ☆715Updated last month
• flashinfer-ai / flashinfer
  FlashInfer: Kernel Library for LLM Serving
  ☆1,143Updated last week
• huggingface / transformers-bloom-inference
  Fast Inference Solutions for BLOOM
  ☆556Updated last month
• mosaicml / streaming
  A Data Streaming Library for Efficient Neural Network Training
  ☆1,077Updated this week
• ray-project / llmperf
  LLMPerf is a library for validating and benchmarking LLMs
  ☆578Updated last month
• hpcaitech / EnergonAI
  Large-scale model inference.
  ☆630Updated last year
• triton-inference-server / tensorrtllm_backend
  The Triton TensorRT-LLM Backend
  ☆654Updated this week
• triton-inference-server / vllm_backend
  ☆170Updated this week
• huggingface / optimum
  🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools
  ☆2,461Updated this week
• S-LoRA / S-LoRA
  S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  ☆1,702Updated 8 months ago
• ray-project / ray-llm
  RayLLM - LLMs on Ray
  ☆1,222Updated 3 months ago
• efeslab / Nanoflow
  A throughput-oriented high-performance serving framework for LLMs
  ☆470Updated this week
• vectorch-ai / ScaleLLM
  A high-performance inference system for large language models, designed for production environments.
  ☆370Updated last week
• bigscience-workshop / Megatron-DeepSpeed
  Ongoing research training transformer language models at scale, including: BERT & GPT-2
  ☆1,317Updated 6 months ago
• triton-inference-server / tutorials
  This repository contains tutorials and examples for Triton Inference Server
  ☆527Updated this week
• databricks / megablocks
  ☆1,167Updated last week
• ninehills / llm-inference-benchmark
  LLM Inference benchmark
  ☆331Updated last month
• ELS-RD / transformer-deploy
  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
  ☆1,643Updated 10 months ago
• DachengLi1 / LongChat
  Official repository for LongChat and LongEval
  ☆505Updated 3 months ago
• triton-inference-server / model_analyzer
  Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Serv…
  ☆419Updated 2 weeks ago
• milvus-io / milvus-lite
  A lightweight version of Milvus
  ☆267Updated 2 weeks ago
• triton-inference-server / client
  Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
  ☆547Updated this week
• WarrenWen666 / AI-Software-Startups
  A Survey of AI startups
  ☆391Updated last year
• hao-ai-lab / LookaheadDecoding
  [ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  ☆1,099Updated 7 months ago
• IST-DASLab / marlin
  FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.
  ☆562Updated 2 weeks ago
• huggingface / text-embeddings-inference
  A blazing fast inference solution for text embeddings models
  ☆2,599Updated this week