huggingface / tokenizersLinks
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
☆10,301Updated 2 weeks ago
Alternatives and similar repositories for tokenizers
Users that are interested in tokenizers are comparing it to the libraries listed below
Sorting:
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆9,377Updated this week
- 🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools☆20,976Updated last week
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,508Updated this week
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,460Updated last month
- Ongoing research training transformer models at scale☆14,602Updated this week
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.☆32,020Updated 2 months ago
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more☆34,299Updated this week
- Development repository for the Triton language and compiler☆17,861Updated this week
- State-of-the-Art Text Embeddings☆18,008Updated this week
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆7,833Updated 6 months ago
- A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…☆16,271Updated this week
- DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.☆41,015Updated this week
- Accessible large language models via k-bit quantization for PyTorch.☆7,815Updated this week
- Trax — Deep Learning with Clear Code and Speed☆8,294Updated 2 months ago
- 🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization…☆3,215Updated last week
- An open-source NLP research library, built on PyTorch.☆11,886Updated 3 years ago
- Simple, safe way to store and distribute tensors☆3,547Updated this week
- ☆2,916Updated last week
- 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal model…☆153,866Updated this week
- Flax is a neural network library for JAX that is designed for flexibility.☆6,977Updated this week
- Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)☆3,005Updated 6 months ago
- Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities☆21,875Updated 5 months ago
- The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!☆8,302Updated this week
- State-of-the-Art Deep Learning scripts organized by models - easy to train and deploy with reproducible accuracy and performance on enter…☆14,630Updated last year
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,687Updated this week
- Papers & presentation materials from Hugging Face's internal science day☆2,053Updated 5 years ago
- Serve, optimize and scale PyTorch models in production☆4,357Updated 4 months ago
- Models, data loaders and abstractions for language processing, powered by PyTorch☆3,560Updated 3 months ago
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,333Updated last month
- Large Language Model Text Generation Inference☆10,709Updated this week