Alternatives and similar repositories for tokenization:

Users that are interested in tokenization are comparing it to the libraries listed below

• Locutusque / TPU-Alignment
  Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free
  ☆230Updated 3 months ago
• abacaj / train-with-fsdp
  ☆92Updated last year
• huggingface / cosmopedia
  ☆496Updated 2 months ago
• jondurbin / bagel
  A bagel, with everything.
  ☆316Updated 10 months ago
• davanstrien / awesome-synthetic-datasets
  awesome synthetic (text) datasets
  ☆259Updated 3 months ago
• VikParuchuri / textbook_quality
  Generate textbook-quality synthetic LLM pretraining data
  ☆494Updated last year
• mlfoundations / evalchemy
  Automatic Evals for LLMs
  ☆201Updated this week
• VikParuchuri / libgen_to_txt
  Convert all of libgen to high quality markdown
  ☆245Updated last year
• huggingface / llm-swarm
  Manage scalable open LLM inference endpoints in Slurm clusters
  ☆252Updated 7 months ago
• tcapelle / llm_recipes
  A set of scripts and notebooks on LLM finetunning and dataset creation
  ☆102Updated 4 months ago
• AblateIt / finetune-study
  Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.
  ☆82Updated last year
• apoorvumang / prompt-lookup-decoding
  ☆501Updated 5 months ago
• Arize-ai / LLMTest_NeedleInAHaystack
  Doing simple retrieval from LLM models at various context lengths to measure accuracy
  ☆100Updated 10 months ago
• sabetAI / BLoRA
  batched loras
  ☆338Updated last year
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 8 months ago
• normster / llm_rules
  RuLES: a benchmark for evaluating rule-following in language models
  ☆217Updated this week
• sdan / selfextend
  an implementation of Self-Extend, to expand the context window via grouped attention
  ☆118Updated last year
• cohere-ai / DiskVectorIndex
  ☆207Updated 7 months ago
• allenai / fm-cheatsheet
  Website for hosting the Open Foundation Models Cheat Sheet.
  ☆262Updated 7 months ago
• teknium1 / LLM-Benchmark-Logs
  Just a bunch of benchmark logs for different LLMs
  ☆119Updated 6 months ago
• Mihaiii / backtrack_sampler
  An easy-to-understand framework for LLM samplers that rewind and revise generated tokens
  ☆129Updated last week
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆75Updated 2 months ago
• JinjieNi / MixEval
  The official evaluation suite and dynamic data release for MixEval.
  ☆231Updated 3 months ago
• srush / GPTWorld
  A puzzle to learn about prompting
  ☆124Updated last year
• euclaise / SlimTrainer
  Full finetuning of large language models without large memory requirements
  ☆93Updated last year
• FastEval / FastEval
  Fast & more realistic evaluation of chat language models. Includes leaderboard.
  ☆183Updated last year
• gautierdag / bpeasy
  Fast bare-bones BPE for modern tokenizer training
  ☆145Updated 3 months ago
• migtissera / Sensei
  Generate Synthetic Data Using OpenAI, MistralAI or AnthropicAI
  ☆222Updated 9 months ago
• DataformerAI / dataformer
  Solving data for LLMs - Create quality synthetic datasets!
  ☆145Updated 3 weeks ago
• FranxYao / Long-Context-Data-Engineering
  Implementation of paper Data Engineering for Scaling Language Models to 128K Context
  ☆451Updated 10 months ago