Alternatives and similar repositories for tokengrams:

Users that are interested in tokengrams are comparing it to the libraries listed below

• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated last year
• jackbandy / bookcorpus-datasheet
  Documentation effort for the BookCorpus dataset
  ☆33Updated 3 years ago
• EleutherAI / best-download
  URL downloader supporting checkpointing and continuous checksumming.
  ☆19Updated last year
• EleutherAI / mdl
  Minimum Description Length probing for neural network representations
  ☆19Updated last month
• laramohan / wikillm
  LLMs as Collaboratively Edited Knowledge Bases
  ☆45Updated last year
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆25Updated last year
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆61Updated 9 months ago
• catid / lllm
  Latent Large Language Models
  ☆17Updated 6 months ago
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated 7 months ago
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆31Updated last year
• mukhal / intrinsic-source-citation
  [COLM '24] Source-Aware Training Enables Knowledge Attribution in Language Models
  ☆17Updated this week
• Zyphra / zcookbook
  Training hybrid models for dummies.
  ☆20Updated 2 months ago
• EleutherAI / improved-t5
  Experiments for efforts to train a new and improved t5
  ☆77Updated 11 months ago
• interview-eval / interview-eval
  Interview-based evaluation of LLMs
  ☆17Updated 2 months ago
• cwhy / rwkv-decon
  Trying to deconstruct RWKV in understandable terms
  ☆14Updated last year
• kaiokendev / cutoff-len-is-context-len
  Demonstration that finetuning RoPE model on larger sequences than the pre-trained model adapts the model context limit
  ☆63Updated last year
• allenai / cached_path
  A file utility for accessing both local and remote files through a unified interface.
  ☆38Updated last week
• leloykun / mmsg
  Generate interleaved text and image content in a structured format you can directly pass to downstream APIs.
  ☆27Updated 5 months ago
• Avmb / inverse_scaling_prize_code_identifier_swap
  Submission to the inverse scaling prize
  ☆23Updated last year
• allenai / adapt-demos
  Lightweight tools for quick and easy LLM demo's
  ☆26Updated 6 months ago
• gonglinyuan / metro_t0
  Code repo for "Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers" (ACL 2023)
  ☆22Updated last year
• argilla-io / distilabel-spin-dibt
  Repository containing the SPIN experiments on the DIBT 10k ranked prompts
  ☆24Updated last year
• iantbutler01 / ditty
  A library for simplifying fine tuning with multi gpu setups in the Huggingface ecosystem.
  ☆16Updated 4 months ago
• Birch-san / booru-embed
  [WIP] Transformer to embed Danbooru labelsets
  ☆13Updated 11 months ago
• haileyschoelkopf / triton-index
  See https://github.com/cuda-mode/triton-index/ instead!
  ☆11Updated 10 months ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆24Updated 3 months ago
• 1rgs / tokenwiz
  A clone of OpenAI's Tokenizer page for HuggingFace Models
  ☆45Updated last year
• mixedbread-ai / binary-embeddings
  Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…
  ☆15Updated 11 months ago
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year