openai / tiktokenLinks
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
☆14,935Updated 3 months ago
Alternatives and similar repositories for tiktoken
Users that are interested in tiktoken are comparing it to the libraries listed below
Sorting:
- LlamaIndex is the leading framework for building LLM-powered agents over your data.☆42,633Updated this week
- the AI-native open-source embedding database☆20,790Updated this week
- DSPy: The framework for programming—not prompting—language models☆26,016Updated this week
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.☆38,770Updated last month
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.☆16,474Updated 6 months ago
- QLoRA: Efficient Finetuning of Quantized LLMs☆10,518Updated last year
- Large Language Model Text Generation Inference☆10,265Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆50,864Updated this week
- Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.☆9,719Updated last year
- Universal LLM Deployment Engine with ML Compilation☆20,880Updated last week
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆42,415Updated 6 months ago
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆18,912Updated this week
- Code and documentation to train Stanford's Alpaca models, and generate the data.☆30,046Updated 11 months ago
- LLMs build upon Evol Insturct: WizardLM, WizardCoder, WizardMath☆9,423Updated 3 weeks ago
- Python bindings for llama.cpp☆9,297Updated this week
- Structured Outputs☆11,990Updated this week
- Semantic cache for LLMs. Fully integrated with LangChain and llama_index.☆7,609Updated 9 months ago
- Letta (formerly MemGPT) is the stateful agents framework with memory, reasoning, and context management.☆17,080Updated this week
- 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.☆12,380Updated this week
- Instruct-tune LLaMA on consumer hardware☆18,927Updated 11 months ago
- 🦜🔗 Build context-aware reasoning applications☆110,444Updated this week
- Examples and guides for using the OpenAI API☆64,951Updated this week
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We als…☆17,551Updated this week
- Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"☆12,153Updated 6 months ago
- Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)☆12,205Updated this week
- Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sag…☆24,658Updated this week
- Tensor library for machine learning☆12,738Updated last week
- Official inference library for Mistral models☆10,320Updated 3 months ago
- The ChatGPT Retrieval Plugin lets you easily find personal or work documents by asking questions in natural language.☆21,191Updated 11 months ago
- [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.☆22,918Updated 10 months ago