google / sentencepieceLinks
Unsupervised text tokenizer for Neural Network-based text generation.
☆11,047Updated this week
Alternatives and similar repositories for sentencepiece
Users that are interested in sentencepiece are comparing it to the libraries listed below
Sorting:
- An open-source NLP research library, built on PyTorch.☆11,857Updated 2 years ago
- Open Source Neural Machine Translation and (Large) Language Models in PyTorch☆6,899Updated 3 months ago
- Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"☆6,383Updated 2 months ago
- Ongoing research training transformer models at scale☆12,701Updated last week
- 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…☆8,896Updated this week
- 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production☆9,859Updated last week
- BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)☆7,513Updated last month
- XLNet: Generalized Autoregressive Pretraining for Language Understanding☆6,183Updated 2 years ago
- Trax — Deep Learning with Clear Code and Speed☆8,225Updated 2 months ago
- Facebook AI Research Sequence-to-Sequence Toolkit written in Python.☆31,582Updated 3 weeks ago
- TensorFlow code and pre-trained models for BERT☆39,286Updated 11 months ago
- Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages☆7,502Updated last week
- A very simple framework for state-of-the-art Natural Language Processing (NLP)☆14,211Updated last week
- 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.☆18,912Updated this week
- Train transformer language models with reinforcement learning.☆14,435Updated this week
- State-of-the-Art Text Embeddings☆17,024Updated 3 weeks ago
- Google AI 2018 BERT pytorch implementation☆6,424Updated last year
- Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conve…☆4,186Updated 2 months ago
- Accessible large language models via k-bit quantization for PyTorch.☆7,192Updated this week
- Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings☆7,057Updated 7 months ago
- Transformer related optimization, including BERT, GPT☆6,224Updated last year
- A library for Multilingual Unsupervised or Supervised word Embeddings☆3,222Updated 2 years ago
- Library for fast text representation and classification.☆26,264Updated last year
- Language-Agnostic SEntence Representations☆3,646Updated last year
- KenLM: Faster and Smaller Language Model Queries☆2,625Updated 3 months ago
- Models, data loaders and abstractions for language processing, powered by PyTorch☆3,542Updated this week
- Fast and memory-efficient exact attention☆18,150Updated this week
- An annotated implementation of the Transformer paper.☆6,324Updated last year
- Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.☆14,519Updated this week
- A tool for extracting plain text from Wikipedia dumps☆3,882Updated last year