openai / tiktokenLinks
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
β15,911Updated 2 weeks ago
Alternatives and similar repositories for tiktoken
Users that are interested in tiktoken are comparing it to the libraries listed below
Sorting:
- Large Language Model Text Generation Inferenceβ10,491Updated last week
- π€ PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.β19,570Updated this week
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We alsβ¦β17,856Updated last week
- LlamaIndex is the leading framework for building LLM-powered agents over your data.β44,256Updated this week
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.β16,931Updated 8 months ago
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.β39,096Updated 3 months ago
- Inference code for Llama modelsβ58,737Updated 7 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMsβ58,035Updated this week
- Universal LLM Deployment Engine with ML Compilationβ21,302Updated last week
- QLoRA: Efficient Finetuning of Quantized LLMsβ10,656Updated last year
- The simplest, fastest repository for training/finetuning medium-sized GPTs.β44,331Updated 9 months ago
- Accessible large language models via k-bit quantization for PyTorch.β7,584Updated this week
- Tensor library for machine learningβ13,134Updated last week
- Code and documentation to train Stanford's Alpaca models, and generate the data.β30,148Updated last year
- Open-source search and retrieval database for AI applications.β23,272Updated this week
- Instruct-tune LLaMA on consumer hardwareβ18,957Updated last year
- Train transformer language models with reinforcement learning.β15,520Updated this week
- π¦π Build context-aware reasoning applications π¦πβ115,290Updated this week
- A guidance language for controlling large language models.β20,681Updated last week
- Unsupervised text tokenizer for Neural Network-based text generation.β11,270Updated this week
- OpenLLaMA, a permissively licensed open source reproduction of Meta AIβs LLaMA 7B trained on the RedPajama datasetβ7,526Updated 2 years ago
- Examples and guides for using the OpenAI APIβ67,901Updated this week
- Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage all AI compute (Kubernetes, 17+ clouds, oβ¦β8,709Updated this week
- Fast and memory-efficient exact attentionβ19,471Updated this week
- Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.β11,758Updated last week
- SGLang is a fast serving framework for large language models and vision language models.β17,823Updated this week
- Python bindings for llama.cppβ9,566Updated last month
- Inference Llama 2 in one file of pure Cβ18,735Updated last year
- πΈ Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloadingβ9,791Updated last year
- Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"β12,660Updated 9 months ago