Alternatives and similar repositories for tokengrams

Users that are interested in tokengrams are comparing it to the libraries listed below

• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆65Updated 4 months ago
• liujch1998 / infini-gram
  ☆69Updated last month
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆104Updated last year
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated 2 years ago
• huggingface / disaggregators
  🤗 Disaggregators: Curated data labelers for in-depth analysis.
  ☆67Updated 2 years ago
• EleutherAI / features-across-time
  Understanding how features learned by neural networks evolve throughout training
  ☆39Updated 11 months ago
• enjalot / latent-sae
  Training code for Sparse Autoencoders on Embedding models
  ☆38Updated 7 months ago
• epfl-dlab / GPTurk
  ☆29Updated 2 years ago
• EleutherAI / best-download
  URL downloader supporting checkpointing and continuous checksumming.
  ☆19Updated last year
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆26Updated 10 months ago
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆77Updated 2 months ago
• AnswerDotAI / ModernBERT-Instruct-mini-cookbook
  ☆49Updated 7 months ago
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆47Updated 2 years ago
• KaiNylund / lm-weights-encode-time
  ☆69Updated last year
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆27Updated 2 years ago
• EleutherAI / improved-t5
  Experiments for efforts to train a new and improved t5
  ☆76Updated last year
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated last month
• bminixhofer / tokenkit
  A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.
  ☆46Updated 2 months ago
• argilla-io / distilabel-spin-dibt
  Repository containing the SPIN experiments on the DIBT 10k ranked prompts
  ☆24Updated last year
• bloomberg / minilmv2.bb
  Our open source implementation of MiniLMv2 (https://aclanthology.org/2021.findings-acl.188)
  ☆61Updated 2 years ago
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆85Updated 3 weeks ago
• deployradiant / pychatml
  Chat Markup Language conversation library
  ☆55Updated last year
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆65Updated last week
• mobiusml / aana_sdk
  Aana SDK is a powerful framework for building AI enabled multimodal applications.
  ☆52Updated last month
• mungg / FABLES
  ☆57Updated last year
• kyutai-labs / dactory
  ☆43Updated this week
• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated 2 years ago
• huggingface / hffs
  **ARCHIVED** Filesystem interface to 🤗 Hub
  ☆58Updated 2 years ago
• haileyschoelkopf / triton-index
  See https://github.com/cuda-mode/triton-index/ instead!
  ☆11Updated last year
• nateraw / spaces-docker-templates
  🚀🤗 A collection of templates for Hugging Face Spaces
  ☆35Updated last year