huggingface / tokenizersLinks
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
☆10,033Updated this week
Alternatives and similar repositories for tokenizers
Users that are interested in tokenizers are comparing it to the libraries listed below
Sorting:
- Unsupervised text tokenizer for Neural Network-based text generation.☆11,221Updated this week
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,415Updated 4 months ago
- State-of-the-Art Text Embeddings☆17,477Updated this week
- 🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools☆20,583Updated last week
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,275Updated 2 weeks ago
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆7,619Updated 3 months ago
- An open-source NLP research library, built on PyTorch.☆11,874Updated 2 years ago
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆9,086Updated last week
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.☆31,767Updated 2 months ago
- Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.☆30,078Updated this week
- Trax — Deep Learning with Clear Code and Speed☆8,267Updated last week
- Papers & presentation materials from Hugging Face's internal science day☆2,048Updated 4 years ago
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,575Updated this week
- XLNet: Generalized Autoregressive Pretraining for Language Understanding☆6,178Updated 2 years ago
- A library for efficient similarity search and clustering of dense vectors.☆36,863Updated last week
- Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conve…☆4,213Updated last week
- 🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization…☆3,066Updated this week
- Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities☆21,681Updated 2 months ago
- Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)☆2,940Updated 2 months ago
- A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…☆15,558Updated this week
- 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal model…☆149,100Updated this week
- A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)☆5,585Updated 4 months ago
- 🤗 Evaluate: A library for easily evaluating machine learning models and datasets.☆2,311Updated 3 weeks ago
- Ongoing research training transformer models at scale☆13,458Updated this week
- Open standard for machine learning interoperability☆19,554Updated this week
- Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the mo…☆22,939Updated last year
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations☆3,273Updated 2 years ago
- Repo for external large-scale work☆6,542Updated last year
- 🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP☆12,742Updated last year
- 🔮 A refreshing functional take on deep learning, compatible with your favorite libraries☆2,878Updated last month