huggingface / tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
☆9,670Updated last month
Alternatives and similar repositories for tokenizers
Users that are interested in tokenizers are comparing it to the libraries listed below
Sorting:
- Unsupervised text tokenizer for Neural Network-based text generation.☆10,878Updated last month
- Ongoing research training transformer models at scale☆12,358Updated this week
- State-of-the-Art Text Embeddings☆16,699Updated this week
- Large Language Model Text Generation Inference☆10,119Updated this week
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆7,389Updated last year
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.☆31,435Updated 4 months ago
- An open-source NLP research library, built on PyTorch.☆11,847Updated 2 years ago
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,461Updated 2 weeks ago
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆8,708Updated this week
- Train transformer language models with reinforcement learning.☆13,703Updated this week
- Development repository for the Triton language and compiler☆15,568Updated this week
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,349Updated 2 weeks ago
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆18,371Updated this week
- A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…☆14,317Updated this week
- Accessible large language models via k-bit quantization for PyTorch.☆7,020Updated this week
- 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools☆20,107Updated this week
- Fast and memory-efficient exact attention☆17,346Updated last week
- A library for efficient similarity search and clustering of dense vectors.☆34,852Updated this week
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,167Updated this week
- Papers & presentation materials from Hugging Face's internal science day☆2,047Updated 4 years ago
- 🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP☆12,665Updated last year
- tiktoken is a fast BPE tokeniser for use with OpenAI's models.☆14,461Updated 2 months ago
- PyTorch original implementation of Cross-lingual Language Model Pretraining.☆2,906Updated 2 years ago
- Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the mo…☆22,866Updated 9 months ago
- Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conve…☆4,178Updated 3 weeks ago
- Transformer related optimization, including BERT, GPT☆6,152Updated last year
- State of the Art Natural Language Processing☆3,965Updated this week
- Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.☆29,473Updated this week
- Models, data loaders and abstractions for language processing, powered by PyTorch☆3,542Updated this week
- Trax — Deep Learning with Clear Code and Speed☆8,204Updated last month