guidance-ai / llgtrtLinks
TensorRT-LLM server with Structured Outputs (JSON) built with Rust
☆57Updated 3 months ago
Alternatives and similar repositories for llgtrt
Users that are interested in llgtrt are comparing it to the libraries listed below
Sorting:
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆192Updated 2 weeks ago
- This repository has code for fine-tuning LLMs with GRPO specifically for Rust Programming using cargo as feedback☆100Updated 4 months ago
- Super-fast Structured Outputs☆350Updated this week
- Faster structured generation☆237Updated 2 months ago
- Official Rust Implementation of Model2Vec☆123Updated last month
- Formatron empowers everyone to control the format of language models' output with minimal overhead.☆220Updated last month
- A high-performance constrained decoding engine based on context free grammar in Rust☆54Updated 2 months ago
- ☆138Updated last year
- Inference engine for GLiNER models, in Rust☆64Updated last month
- Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.☆409Updated this week
- The Easiest Rust Interface for Local LLMs and an Interface for Deterministic Signals from Probabilistic LLM Vibes☆219Updated last week
- ☆130Updated last year
- implement llava using candle☆15Updated last year
- A simple, CUDA or CPU powered, library for creating vector embeddings using Candle and models from Hugging Face☆37Updated last year
- A python package for serving LLM on OpenAI-compatible API endpoints with prompt caching using MLX.☆90Updated last month
- Inference Llama 2 in one file of zero-dependency, zero-unsafe Rust☆38Updated 2 years ago
- 🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.☆137Updated last year
- A Pure Rust based LLM (Any LLM based MLLM such as Spark-TTS) Inference Engine, powering by Candle framework.☆144Updated last week
- Fast serverless LLM inference, in Rust.☆88Updated 5 months ago
- Super-simple, fully Rust powered "memory" (doc store + semantic search) for LLM projects, semantic search, etc.☆62Updated last year
- Guaranteed Structured Output from any Language Model via Hierarchical State Machines☆141Updated 2 months ago
- Unofficial python bindings for the rust llm library. 🐍❤️🦀☆75Updated last year
- Structured outputs for LLMs☆49Updated last year
- A single-binary, GPU-accelerated LLM server (HTTP and WebSocket API) written in Rust☆80Updated last year
- OpenAI compatible API for serving LLAMA-2 model☆218Updated last year
- ☆388Updated this week
- ☆66Updated last year
- Built for demanding AI workflows, this gateway offers low-latency, provider-agnostic access, ensuring your AI applications run smoothly a…☆73Updated 2 months ago
- A Fish Speech implementation in Rust, with Candle.rs☆94Updated 2 months ago
- Rust implementation of Surya☆58Updated 5 months ago