huggingface / tokenizers

Related projects ⓘ

Alternatives and complementary repositories for tokenizers

• google / sentencepiece
  Unsupervised text tokenizer for Neural Network-based text generation.
  ☆10,295Updated 2 weeks ago
• NVIDIA / Megatron-LM
  Ongoing research training transformer models at scale
  ☆10,595Updated this week
• google-research / text-to-text-transfer-transformer
  Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"
  ☆6,178Updated 2 months ago
• allenai / allennlp
  An open-source NLP research library, built on PyTorch.
  ☆11,759Updated last year
• facebookresearch / fairseq
  Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
  ☆30,552Updated last month
• UKPLab / sentence-transformers
  State-of-the-Art Text Embeddings
  ☆15,368Updated this week
• facebookresearch / faiss
  A library for efficient similarity search and clustering of dense vectors.
  ☆31,488Updated this week
• huggingface / accelerate
  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (i…
  ☆7,958Updated this week
• google / trax
  Trax — Deep Learning with Clear Code and Speed
  ☆8,101Updated 2 months ago
• OpenNMT / OpenNMT-py
  Open Source Neural Machine Translation and (Large) Language Models in PyTorch
  ☆6,773Updated 4 months ago
• PAIR-code / lit
  The Learning Interpretability Tool: Interactively analyze ML models to understand their behavior in an extensible and framework agnostic …
  ☆3,492Updated 2 weeks ago
• huggingface / trl
  Train transformer language models with reinforcement learning.
  ☆10,086Updated this week
• jessevig / bertviz
  BertViz: Visualize Attention in NLP Models (BERT, GPT2, BART, etc.)
  ☆6,952Updated last year
• NVIDIA / NeMo
  A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Auto…
  ☆12,150Updated this week
• zihangdai / xlnet
  XLNet: Generalized Autoregressive Pretraining for Language Understanding
  ☆6,182Updated last year
• facebookresearch / XLM
  PyTorch original implementation of Cross-lingual Language Model Pretraining.
  ☆2,892Updated last year
• flairNLP / flair
  A very simple framework for state-of-the-art Natural Language Processing (NLP)
  ☆13,946Updated this week
• facebookresearch / LASER
  Language-Agnostic SEntence Representations
  ☆3,600Updated 6 months ago
• huggingface / peft
  🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
  ☆16,471Updated this week
• huggingface / awesome-papers
  Papers & presentation materials from Hugging Face's internal science day
  ☆2,037Updated 4 years ago
• huggingface / transformers
  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
  ☆135,166Updated this week
• microsoft / onnxruntime
  ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
  ☆14,722Updated this week
• facebookresearch / pytext
  A natural language modeling framework based on PyTorch
  ☆6,338Updated 2 years ago
• Dao-AILab / flash-attention
  Fast and memory-efficient exact attention
  ☆14,279Updated this week
• google-research / albert
  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  ☆3,247Updated last year
• triton-lang / triton
  Development repository for the Triton language and compiler
  ☆13,443Updated this week
• doccano / doccano
  Open source annotation tool for machine learning practitioners.
  ☆9,586Updated this week
• microsoft / DeBERTa
  The implementation of DeBERTa
  ☆1,991Updated last year
• snorkel-team / snorkel
  A system for quickly generating training data with weak supervision
  ☆5,812Updated 6 months ago
• karpathy / minGPT
  A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training
  ☆20,199Updated 3 months ago