openai / tiktokenLinks
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
โ16,330Updated 3 weeks ago
Alternatives and similar repositories for tiktoken
Users that are interested in tiktoken are comparing it to the libraries listed below
Sorting:
- ๐ค PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.โ19,900Updated this week
- Examples and guides for using the OpenAI APIโ68,869Updated this week
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We alsโฆโ17,979Updated this week
- The simplest, fastest repository for training/finetuning medium-sized GPTs.โ47,749Updated 10 months ago
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.โ17,160Updated 10 months ago
- Open-source search and retrieval database for AI applications.โ24,045Updated this week
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.โ39,185Updated 4 months ago
- Inference code for Llama modelsโ58,873Updated 9 months ago
- LlamaIndex is the leading framework for building LLM-powered agents over your data.โ44,895Updated this week
- [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.โ23,822Updated last year
- ๐ฆ๐ Build context-aware reasoning applicationsโ117,729Updated last week
- Python bindings for llama.cppโ9,678Updated 2 months ago
- Tensor library for machine learningโ13,332Updated this week
- Large Language Model Text Generation Inferenceโ10,580Updated last month
- A programming framework for agentic AIโ51,154Updated 3 weeks ago
- QLoRA: Efficient Finetuning of Quantized LLMsโ10,710Updated last year
- The official Python library for the OpenAI APIโ29,041Updated this week
- Instruct-tune LLaMA on consumer hardwareโ18,977Updated last year
- Build resilient language agents as graphs.โ20,185Updated this week
- Code and documentation to train Stanford's Alpaca models, and generate the data.โ30,190Updated last year
- The official repo of Qwen (้ไนๅ้ฎ) chat & pretrained large language model proposed by Alibaba Cloud.โ19,579Updated 3 weeks ago
- Unsupervised text tokenizer for Neural Network-based text generation.โ11,394Updated 3 weeks ago
- Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"โ12,843Updated 10 months ago
- Inference Llama 2 in one file of pure Cโ18,872Updated last year
- The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.โ8,775Updated last year
- A high-throughput and memory-efficient inference and serving engine for LLMsโ60,980Updated this week
- Universal LLM Deployment Engine with ML Compilationโ21,527Updated this week
- RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable)โฆโ14,065Updated last week
- Train transformer language models with reinforcement learning.โ16,012Updated this week
- Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adโฆโ6,079Updated 3 months ago