huggingface / tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
☆9,038Updated this week
Related projects ⓘ
Alternatives and complementary repositories for tokenizers
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,160Updated last month
- Unsupervised text tokenizer for Neural Network-based text generation.☆10,252Updated last week
- An open-source NLP research library, built on PyTorch.☆11,756Updated last year
- Trax — Deep Learning with Clear Code and Speed☆8,093Updated last month
- TensorFlow code and pre-trained models for BERT☆38,156Updated 3 months ago
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆7,911Updated this week
- Serve, optimize and scale PyTorch models in production☆4,209Updated 2 weeks ago
- Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.☆28,337Updated this week
- State-of-the-Art Text Embeddings☆15,236Updated this week
- Ongoing research training transformer models at scale☆10,480Updated this week
- Papers & presentation materials from Hugging Face's internal science day☆2,035Updated 4 years ago
- Train transformer language models with reinforcement learning.☆9,967Updated this week
- 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools☆19,228Updated this week
- Development repository for the Triton language and compiler☆13,311Updated this week
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,284Updated last week
- A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…☆12,054Updated this week
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more☆30,426Updated this week
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆13,928Updated 2 weeks ago
- Open source annotation tool for machine learning practitioners.☆9,549Updated last week
- Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conve…☆4,101Updated 5 months ago
- XLNet: Generalized Autoregressive Pretraining for Language Understanding☆6,181Updated last year
- Flax is a neural network library for JAX that is designed for flexibility.☆6,101Updated this week
- Fast and memory-efficient exact attention☆14,109Updated this week
- A library for efficient similarity search and clustering of dense vectors.☆31,320Updated this week
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆6,919Updated last year
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆16,354Updated this week
- A natural language modeling framework based on PyTorch☆6,338Updated 2 years ago
- 🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools☆2,556Updated this week
- The Learning Interpretability Tool: Interactively analyze ML models to understand their behavior in an extensible and framework agnostic …☆3,488Updated this week
- ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator☆14,642Updated this week