dusty-nv / NanoLLM
Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.
☆255Updated 6 months ago
Alternatives and similar repositories for NanoLLM:
Users that are interested in NanoLLM are comparing it to the libraries listed below
- A reference application for a local AI assistant with LLM and RAG☆109Updated 4 months ago
- ☆107Updated 2 weeks ago
- A tutorial introducing knowledge distillation as an optimization technique for deployment on NVIDIA Jetson☆190Updated last year
- A collection of reference AI microservices and workflows for Jetson Platform Services☆38Updated 2 months ago
- A project that optimizes Whisper for low latency inference using NVIDIA TensorRT☆79Updated 6 months ago
- A utility library to help integrate Python applications with Metropolis Microservices for Jetson☆12Updated 4 months ago
- Collection of reference workflows for building intelligent agents with NIMs☆152Updated 3 months ago
- Zero-copy multimodal vector DB with CUDA and CLIP/SigLIP☆54Updated 10 months ago
- Quick start scripts and tutorial notebooks to get started with TAO Toolkit☆78Updated 7 months ago
- Using FastChat-T5 Large Language Model, Vosk API for automatic speech recognition, and Piper for text-to-speech☆119Updated last year
- This repo has the code of the 3 demos I presented at Google Gemma2 DevDay Tokyo, using Gemma2 on a Jetson Orin Nano device.☆40Updated last week
- A project that optimizes OWL-ViT for real-time inference with NVIDIA TensorRT.☆320Updated 2 months ago
- ☆94Updated 7 months ago
- A reference example for integrating NanoOwl with Metropolis Microservices for Jetson☆30Updated 10 months ago
- Blueprint for Ingesting massive volumes of live or archived videos and extract insights for summarization and interactive Q&A☆30Updated last month
- ASR/NLP/TTS deep learning inference library for NVIDIA Jetson using PyTorch and TensorRT☆205Updated last year
- ☆17Updated 2 weeks ago
- A repository for affordable, easy-to-assemble robot arms designed for teleoperation applications.☆78Updated 2 months ago
- The jetson-examples repository by Seeed Studio offers a seamless, one-line command deployment to run vision AI and Generative AI models o…☆168Updated 2 months ago
- ☆172Updated last year
- TAO Toolkit deep learning networks with PyTorch backend☆91Updated 5 months ago
- This reference can be used with any existing OpenAI integrated apps to run with TRT-LLM inference locally on GeForce GPU on Windows inste…☆120Updated last year
- A family of compressed models obtained via pruning and knowledge distillation☆334Updated 5 months ago
- From scratch implementation of a vision language model in pure PyTorch☆212Updated 11 months ago
- Awesome Mobile LLMs☆166Updated 3 weeks ago
- The NVIDIA RTX™ AI Toolkit is a suite of tools and SDKs for Windows developers to customize, optimize, and deploy AI models across RTX PC…☆148Updated 5 months ago
- An innovative library for efficient LLM inference via low-bit quantization☆352Updated 7 months ago
- Self-host LLMs with vLLM and BentoML☆105Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆262Updated 6 months ago
- Python bindings for ggml☆140Updated 7 months ago