Alternatives and similar repositories for tokenization:

Users that are interested in tokenization are comparing it to the libraries listed below

• huggingface / llm-swarm
  Manage scalable open LLM inference endpoints in Slurm clusters
  ☆254Updated 9 months ago
• Locutusque / TPU-Alignment
  Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free
  ☆231Updated 6 months ago
• VikParuchuri / textbook_quality
  Generate textbook-quality synthetic LLM pretraining data
  ☆498Updated last year
• davanstrien / awesome-synthetic-datasets
  awesome synthetic (text) datasets
  ☆278Updated 6 months ago
• abacaj / train-with-fsdp
  ☆92Updated last year
• jondurbin / bagel
  A bagel, with everything.
  ☆320Updated last year
• taylorai / galactic
  data cleaning and curation for unstructured text
  ☆329Updated 9 months ago
• allenai / fm-cheatsheet
  Website for hosting the Open Foundation Models Cheat Sheet.
  ☆267Updated 2 weeks ago
• center-for-humans-and-machines / transformer-heads
  Toolkit for attaching, training, saving and loading of new heads for transformer models
  ☆276Updated 2 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 10 months ago
• huggingface / data-is-better-together
  Let's build better datasets, together!
  ☆259Updated 4 months ago
• tcapelle / llm_recipes
  A set of scripts and notebooks on LLM finetunning and dataset creation
  ☆107Updated 7 months ago
• teknium1 / LLM-Benchmark-Logs
  Just a bunch of benchmark logs for different LLMs
  ☆119Updated 9 months ago
• lm-sys / llm-decontaminator
  Code for the paper "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples"
  ☆301Updated last year
• AnswerDotAI / fastdata
  ☆151Updated 5 months ago
• cohere-ai / DiskVectorIndex
  ☆210Updated 10 months ago
• VikParuchuri / libgen_to_txt
  Convert all of libgen to high quality markdown
  ☆251Updated last year
• AblateIt / finetune-study
  Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.
  ☆82Updated last year
• cognitivecomputations / laserRMT
  This is our own implementation of 'Layer Selective Rank Reduction'
  ☆237Updated 11 months ago
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆136Updated 9 months ago
• apoorvumang / prompt-lookup-decoding
  ☆529Updated 8 months ago
• JinjieNi / MixEval
  The official evaluation suite and dynamic data release for MixEval.
  ☆238Updated 5 months ago
• daniel-furman / sft-demos
  Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
  ☆76Updated 6 months ago
• huggingface / cosmopedia
  ☆515Updated 5 months ago
• huggingface / text-clustering
  Easily embed, cluster and semantically label text datasets
  ☆530Updated last year
• Muhtasham / summarization-eval
  📝 Reference-Free automatic summarization evaluation with potential hallucination detection
  ☆100Updated last year
• cognitivecomputations / grokadamw
  ☆129Updated 8 months ago
• srush / GPTWorld
  A puzzle to learn about prompting
  ☆127Updated last year
• NousResearch / Open-Reasoning-Tasks
  A comprehensive repository of reasoning tasks for LLMs (and beyond)
  ☆437Updated 7 months ago
• euclaise / SlimTrainer
  Full finetuning of large language models without large memory requirements
  ☆94Updated last year