openai / tiktokenLinks
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
☆16,561Updated last month
Alternatives and similar repositories for tiktoken
Users that are interested in tiktoken are comparing it to the libraries listed below
Sorting:
- Open-source search and retrieval database for AI applications.☆24,388Updated this week
- LlamaIndex is the leading framework for building LLM-powered agents over your data.☆45,244Updated this week
- Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.☆17,277Updated 2 weeks ago
- A guidance language for controlling large language models.☆20,911Updated last month
- Examples and guides for using the OpenAI API☆69,211Updated this week
- 🦜🔗 The platform for reliable agents.☆119,538Updated this week
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,441Updated last week
- DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.☆40,670Updated last week
- Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Also available in the cl…☆27,100Updated this week
- Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We als…☆18,033Updated 2 weeks ago
- Build and share delightful machine learning apps, all in Python. 🌟 Star to support our work!☆40,534Updated this week
- Large Language Model Text Generation Inference☆10,643Updated last week
- An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.☆39,250Updated 5 months ago
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆20,050Updated this week
- Weaviate is an open-source vector database that stores both objects and vectors, allowing for the combination of vector search with struc…☆15,053Updated this week
- The ChatGPT Retrieval Plugin lets you easily find personal or work documents by asking questions in natural language.☆21,234Updated last year
- Train transformer language models with reinforcement learning.☆16,308Updated this week
- OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama dataset☆7,524Updated 2 years ago
- The official Python library for the OpenAI API☆29,277Updated this week
- QLoRA: Efficient Finetuning of Quantized LLMs☆10,744Updated last year
- Fast and memory-efficient exact attention☆20,541Updated this week
- Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"☆12,945Updated 11 months ago
- 🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.☆31,617Updated this week
- Code and documentation to train Stanford's Alpaca models, and generate the data.☆30,218Updated last year
- A library for efficient similarity search and clustering of dense vectors.☆37,969Updated this week
- A high-throughput and memory-efficient inference and serving engine for LLMs☆63,144Updated this week
- 📋 A list of open LLMs available for commercial use.☆12,509Updated 9 months ago
- DSPy: The framework for programming—not prompting—language models☆30,008Updated this week
- [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.☆23,957Updated last year
- Inference code for Llama models☆58,924Updated 9 months ago