openai / tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
☆11,796Updated last month
Related projects: ⓘ
- the AI-native open-source embedding database☆14,555Updated this week
- LlamaIndex is a data framework for your LLM applications☆35,450Updated this week
- A guidance language for controlling large language models.☆18,698Updated this week
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.☆14,687Updated 3 weeks ago
- Instruct-tune LLaMA on consumer hardware☆18,537Updated last month
- Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform☆10,812Updated last year
- QLoRA: Efficient Finetuning of Quantized LLMs☆9,906Updated 3 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆26,822Updated this week
- Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)☆11,200Updated this week
- Large Language Model Text Generation Inference☆8,762Updated this week
- Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datase…☆11,582Updated last week
- Code and documentation to train Stanford's Alpaca models, and generate the data.☆29,346Updated 2 months ago
- DSPy: The framework for programming—not prompting—foundation models☆16,773Updated this week
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.☆36,446Updated this week
- OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama dataset☆7,339Updated last year
- LLMs build upon Evol Insturct: WizardLM, WizardCoder, WizardMath☆9,194Updated last month
- The ChatGPT Retrieval Plugin lets you easily find personal or work documents by asking questions in natural language.☆21,016Updated 2 months ago
- Official inference library for Mistral models☆9,514Updated this week
- Inference Llama 2 in one file of pure C☆17,153Updated last month
- 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.☆9,780Updated this week
- Universal LLM Deployment Engine with ML Compilation☆18,642Updated this week
- Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Ad…☆5,958Updated last week
- Python bindings for llama.cpp☆7,723Updated this week
- Inference code for CodeLlama models☆15,861Updated last month
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆15,839Updated this week
- StableLM: Stability AI Language Models☆15,842Updated 5 months ago
- Tensor library for machine learning☆10,846Updated last week
- 📋 A list of open LLMs available for commercial use.☆10,912Updated 2 months ago
- 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading☆9,079Updated last week
- Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.☆9,731Updated last week