π₯ Fast State-of-the-Art Tokenizers optimized for Research and Production
β11,270Nov 20, 2026Updated this week
Alternatives and similar repositories for tokenizers
Users that are interested in tokenizers are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- π€ The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation toolsβ22,178Updated this week
- Unsupervised text tokenizer for Neural Network-based text generation.β12,174Updated this week
- π€ Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal modelβ¦β169,344Updated this week
- π A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (iβ¦β9,952Updated this week
- Minimalist ML framework for Rustβ21,370Updated this week
- GPU virtual machines on DigitalOcean Gradient AI β’ AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)β3,091Jan 13, 2026Updated 10 months ago
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.β32,206Sep 30, 2025Updated last year
- State-of-the-Art Embeddings, Retrieval, and Rerankingβ19,297Updated this week
- A very simple framework for state-of-the-art Natural Language Processing (NLP)β14,398Oct 19, 2026Updated last month
- An open-source NLP research library, built on PyTorch.β11,882Nov 22, 2022Updated 3 years ago
- Pretrain, finetune ANY AI model of ANY size on 1 or 10,000+ GPUs with zero code changes.β31,468Updated this week
- A library for efficient similarity search and clustering of dense vectors.β41,842Updated this week
- Papers & presentation materials from Hugging Face's internal science dayβ2,054Oct 31, 2020Updated 6 years ago
- π Accelerate inference and training of π€ Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimizationβ¦β3,525Updated this week
- Wordpress hosting with auto-scaling - Free Trial Offer β’ AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Simple, safe way to store and distribute tensorsβ3,974Updated this week
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and moreβ36,665Updated this week
- Extremely fast Query Engine for DataFrames, written in Rustβ40,417Updated this week
- DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.β43,347Updated this week
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"β6,564Oct 12, 2026Updated last month
- tiktoken is a fast BPE tokeniser for use with OpenAI's models.β19,700Nov 8, 2026Updated last week
- Rust bindings for the C++ api of PyTorch.β5,520Nov 10, 2026Updated last week
- π« Industrial-strength Natural Language Processing (NLP) in Pythonβ34,082Updated this week
- Qdrant - High-performance, massive-scale Vector Database and Vector Search Engine for the next generation of AI. Also available in the clβ¦β35,653Updated this week
- AI Agents on DigitalOcean Gradient AI Platform β’ AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- Burn is a next generation tensor library and Deep Learning Framework that doesn't compromise on flexibility, efficiency and portability.β16,319Updated this week
- Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rustβ16,463Updated this week
- π€ PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.β21,912Updated this week
- TensorFlow code and pre-trained models for BERTβ40,019Jul 23, 2024Updated 2 years ago
- Large Language Model Text Generation Inferenceβ10,876Mar 21, 2026Updated 8 months ago
- πͺβKnock Knock: Get notified when your training ends with only two additional lines of codeβ2,824Jun 23, 2023Updated 3 years ago
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languagesβ7,920Updated this week
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generatorsβ2,366Mar 23, 2024Updated 2 years ago
- Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.β44,352Updated this week
- GPU virtual machines on DigitalOcean Gradient AI β’ AdGet to production fast with high-performance AMD and NVIDIA GPUs you can spin up in seconds. The definition of operational simplicity.
- Train transformer language models with reinforcement learning.β19,788Updated this week
- Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalitiesβ22,247Sep 21, 2026Updated 2 months ago
- A natural language modeling framework based on PyTorchβ6,284Oct 17, 2022Updated 4 years ago
- Rust bindings for the Python interpreterβ16,348Updated this week
- π Scalable embedding, reasoning, ranking for images and sentences with CLIPβ12,841Jan 23, 2024Updated 2 years ago
- Ongoing research training transformer models at scaleβ18,451Updated this week
- Models, data loaders and abstractions for language processing, powered by PyTorchβ3,552Sep 10, 2025Updated last year
Are these results useful?