wangcx18 / llm-vscode-inference-serverLinks
An endpoint server for efficiently serving quantized open-source LLMs for code.
☆56Updated last year
Alternatives and similar repositories for llm-vscode-inference-server
Users that are interested in llm-vscode-inference-server are comparing it to the libraries listed below
Sorting:
- Some simple scripts that I use day-to-day when working with LLMs and Huggingface Hub☆162Updated last year
- starcoder server for huggingface-vscdoe custom endpoint☆173Updated last year
- Visual Studio Code extension for WizardCoder☆149Updated 2 years ago
- ☆199Updated last year
- ☆55Updated 2 years ago
- Deploy your GGML models to HuggingFace Spaces with Docker and gradio☆37Updated 2 years ago
- A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.☆64Updated last year
- Falcon LLM ggml framework with CPU and GPU support☆247Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMs☆52Updated last year
- An OpenAI-like LLaMA inference API☆113Updated last year
- Host the GPTQ model using AutoGPTQ as an API that is compatible with text generation UI API.☆91Updated 2 years ago
- ☆161Updated 3 weeks ago
- ☆63Updated 5 months ago
- An OpenAI API compatible LLM inference server based on ExLlamaV2.☆25Updated last year
- The RunPod worker template for serving our large language model endpoints. Powered by vLLM.☆357Updated last week
- cli tool to quantize gguf, gptq, awq, hqq and exl2 models☆75Updated 8 months ago
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆126Updated last year
- Examples of models deployable with Truss☆196Updated last week
- Convenient wrapper for fine-tuning and inference of Large Language Models (LLMs) with several quantization techniques (GTPQ, bitsandbytes…☆145Updated last year
- An OpenAI Completions API compatible server for NLP transformers models☆65Updated last year
- Locally running LLM with internet access☆96Updated last month
- Low-Rank adapter extraction for fine-tuned transformers models☆175Updated last year
- Landmark Attention: Random-Access Infinite Context Length for Transformers QLoRA☆123Updated 2 years ago
- LLaMA Server combines the power of LLaMA C++ with the beauty of Chatbot UI.☆129Updated 2 years ago
- Lord of LLMS☆294Updated last month
- ☆67Updated last year
- Python bindings for llama.cpp☆66Updated last year
- Self-host LLMs with vLLM and BentoML☆140Updated this week
- A fast batching API to serve LLM models☆185Updated last year
- OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)☆276Updated last year