Alternatives and similar repositories for tokengrams:

Users that are interested in tokengrams are comparing it to the libraries listed below

• Zyphra / zcookbook
  Training hybrid models for dummies.
  ☆20Updated last month
• EleutherAI / mdl
  Minimum Description Length probing for neural network representations
  ☆19Updated last month
• EleutherAI / best-download
  URL downloader supporting checkpointing and continuous checksumming.
  ☆19Updated last year
• Avmb / inverse_scaling_prize_code_identifier_swap
  Submission to the inverse scaling prize
  ☆23Updated last year
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆31Updated last year
• EleutherAI / semantic-memorization
  ☆44Updated 3 months ago
• facebookresearch / DIG-In
  This library supports evaluating disparities in generated image quality, diversity, and consistency between geographic regions.
  ☆20Updated 9 months ago
• catid / lllm
  Latent Large Language Models
  ☆17Updated 6 months ago
• iantbutler01 / ditty
  A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.
  ☆16Updated 4 months ago
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆46Updated last year
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆46Updated last year
• haileyschoelkopf / triton-index
  See https://github.com/cuda-mode/triton-index/ instead!
  ☆11Updated 10 months ago
• AlexWan0 / infini-gram
  An unofficial implementation of the Infini-gram model proposed by Liu et al. (2024)
  ☆30Updated 8 months ago
• jackbandy / bookcorpus-datasheet
  Documentation effort for the BookCorpus dataset
  ☆33Updated 3 years ago
• TheoCoombes / crawlingathome
  A client library for LAION's effort to filter CommonCrawl with CLIP, building a large scale image-text dataset.
  ☆32Updated last year
• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated last year
• HazyResearch / aioli
  Aioli: A unified optimization framework for language model data mixing
  ☆22Updated last month
• allenai / adapt-demos
  Lightweight tools for quick and easy LLM demo's
  ☆26Updated 5 months ago
• google-research-datasets / QAmeleon
  QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…
  ☆34Updated last year
• huggingface / gaia
  Hugging Face and Pyserini interoperability
  ☆20Updated last year
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 3 months ago
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆61Updated 9 months ago
• gonglinyuan / metro_t0
  Code repo for "Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers" (ACL 2023)
  ☆22Updated last year
• ConiferLabsWA / flan-ul2-alpaca
  ☆32Updated last year
• nbroad1881 / strideformer
  Using short models to classify long texts
  ☆21Updated 2 years ago
• srush / drop7
  ☆18Updated 10 months ago
• mukhal / intrinsic-source-citation
  [COLM '24] Source-Aware Training Enables Knowledge Attribution in Language Models
  ☆17Updated 7 months ago
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year