openai / tiktokenLinks
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
☆16,097Updated last week
Alternatives and similar repositories for tiktoken
Users that are interested in tiktoken are comparing it to the libraries listed below
Sorting:
- LlamaIndex is the leading framework for building LLM-powered agents over your data.☆44,575Updated this week
- A guidance language for controlling large language models.☆20,817Updated last week
- Large Language Model Text Generation Inference☆10,550Updated 3 weeks ago
- 🦜🔗 Build context-aware reasoning applications☆116,801Updated this week
- Train transformer language models with reinforcement learning.☆15,739Updated this week
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆19,743Updated this week
- Instruct-tune LLaMA on consumer hardware☆18,958Updated last year
- The simplest, fastest repository for training/finetuning medium-sized GPTs.☆44,823Updated 9 months ago
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.☆39,144Updated 4 months ago
- QLoRA: Efficient Finetuning of Quantized LLMs☆10,680Updated last year
- Open-source search and retrieval database for AI applications.☆23,665Updated this week
- Tensor library for machine learning☆13,230Updated last week
- Python bindings for llama.cpp☆9,635Updated last month
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.☆17,061Updated 9 months ago
- A high-throughput and memory-efficient inference and serving engine for LLMs☆59,413Updated this week
- Code and documentation to train Stanford's Alpaca models, and generate the data.☆30,166Updated last year
- Build and share delightful machine learning apps, all in Python. 🌟 Star to support our work!☆40,109Updated this week
- Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Ad…☆6,076Updated 3 months ago
- Accessible large language models via k-bit quantization for PyTorch.☆7,627Updated this week
- DSPy: The framework for programming—not prompting—language models☆28,825Updated this week
- Semantic cache for LLMs. Fully integrated with LangChain and llama_index.☆7,783Updated 2 months ago
- The ChatGPT Retrieval Plugin lets you easily find personal or work documents by asking questions in natural language.☆21,229Updated last year
- Go ahead and axolotl questions☆10,551Updated this week
- Examples and guides for using the OpenAI API☆68,241Updated this week
- LLM inference in C/C++☆87,149Updated last week
- Inference Llama 2 in one file of pure C☆18,801Updated last year
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,331Updated this week
- Universal LLM Deployment Engine with ML Compilation☆21,440Updated this week
- Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean…☆12,835Updated last week
- Open-source vector similarity search for Postgres☆17,803Updated last week