huggingface / tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
☆9,627Updated last week
Alternatives and similar repositories for tokenizers:
Users that are interested in tokenizers are comparing it to the libraries listed below
- An open-source NLP research library, built on PyTorch.☆11,843Updated 2 years ago
- State-of-the-Art Text Embeddings☆16,554Updated this week
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,341Updated 2 months ago
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆7,344Updated last year
- Unsupervised text tokenizer for Neural Network-based text generation.☆10,816Updated 3 weeks ago
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,148Updated this week
- 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools☆20,007Updated last week
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆8,645Updated this week
- A natural language modeling framework based on PyTorch☆6,325Updated 2 years ago
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,441Updated last week
- XLNet: Generalized Autoregressive Pretraining for Language Understanding☆6,187Updated last year
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.☆31,359Updated 3 months ago
- Train transformer language models with reinforcement learning.☆13,373Updated this week
- 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.☆143,383Updated this week
- Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the mo…☆22,841Updated 9 months ago
- Trax — Deep Learning with Clear Code and Speed☆8,200Updated 2 weeks ago
- A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…☆13,734Updated this week
- A library for efficient similarity search and clustering of dense vectors.☆34,564Updated this week
- Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conve…☆4,172Updated this week
- Ongoing research training transformer models at scale☆12,164Updated this week
- TensorFlow code and pre-trained models for BERT☆39,071Updated 9 months ago
- A framework for training and evaluating AI models on a variety of openly available dialogue datasets.☆10,527Updated last year
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators☆2,352Updated last year
- Natural Language Processing Best Practices & Examples☆6,407Updated 2 years ago
- PyTorch original implementation of Cross-lingual Language Model Pretraining.☆2,906Updated 2 years ago
- A system for quickly generating training data with weak supervision☆5,852Updated 11 months ago
- Fast and memory-efficient exact attention☆17,032Updated this week
- Code for the paper "Language Models are Unsupervised Multitask Learners"☆23,360Updated 8 months ago
- Open Source Neural Machine Translation and (Large) Language Models in PyTorch☆6,866Updated last month
- 🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP☆12,642Updated last year