guidance-ai / llgtrtLinks
TensorRT-LLM server with Structured Outputs (JSON) built with Rust
☆58Updated 4 months ago
Alternatives and similar repositories for llgtrt
Users that are interested in llgtrt are comparing it to the libraries listed below
Sorting:
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆199Updated last month
- A high-performance constrained decoding engine based on context free grammar in Rust☆55Updated 3 months ago
- Faster structured generation☆245Updated 3 months ago
- This repository has code for fine-tuning LLMs with GRPO specifically for Rust Programming using cargo as feedback☆101Updated 5 months ago
- Super-fast Structured Outputs☆432Updated last week
- Official Rust Implementation of Model2Vec☆124Updated last month
- Fast serverless LLM inference, in Rust.☆88Updated 5 months ago
- Formatron empowers everyone to control the format of language models' output with minimal overhead.☆223Updated 2 months ago
- Inference engine for GLiNER models, in Rust☆64Updated last month
- A simple, CUDA or CPU powered, library for creating vector embeddings using Candle and models from Hugging Face☆37Updated last year
- implement llava using candle☆15Updated last year
- ☆130Updated last year
- The Easiest Rust Interface for Local LLMs and an Interface for Deterministic Signals from Probabilistic LLM Vibes☆229Updated 2 weeks ago
- Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.☆62Updated last year
- ☆138Updated last year
- Inference Llama 2 in one file of zero-dependency, zero-unsafe Rust☆38Updated 2 years ago
- Guaranteed Structured Output from any Language Model via Hierarchical State Machines☆145Updated 2 months ago
- A python package for serving LLM on OpenAI-compatible API endpoints with prompt caching using MLX.☆92Updated last month
- ☆33Updated 9 months ago
- Pivotal Token Search☆120Updated last month
- A single-binary, GPU-accelerated LLM server (HTTP and WebSocket API) written in Rust☆80Updated last year
- Structured outputs for LLMs☆50Updated last year
- A Pure Rust based LLM (Any LLM based MLLM such as Spark-TTS) Inference Engine, powering by Candle framework.☆153Updated last month
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆139Updated last year
- Built for demanding AI workflows, this gateway offers low-latency, provider-agnostic access, ensuring your AI applications run smoothly a…☆73Updated 2 months ago
- Unofficial python bindings for the rust llm library. 🐍❤️🦀☆75Updated 2 years ago
- ☆12Updated 6 months ago
- Unofficial Rust bindings to Apple's mlx framework☆184Updated last week
- OpenAI compatible API for serving LLAMA-2 model☆218Updated last year
- ☆199Updated last year