SumanthRH / tokenizationLinks
A comprehensive deep dive into the world of tokens
☆226Updated last year
Alternatives and similar repositories for tokenization
Users that are interested in tokenization are comparing it to the libraries listed below
Sorting:
- Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free☆231Updated 11 months ago
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆195Updated last year
- Fast bare-bones BPE for modern tokenizer training☆167Updated 4 months ago
- A puzzle to learn about prompting☆135Updated 2 years ago
- Website for hosting the Open Foundation Models Cheat Sheet.☆267Updated 5 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated 2 years ago
- Generate textbook-quality synthetic LLM pretraining data☆505Updated 2 years ago
- experiments with inference on llama☆103Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆273Updated last year
- A set of scripts and notebooks on LLM finetunning and dataset creation☆110Updated last year
- ☆94Updated 2 years ago
- An introduction to LLM Sampling☆79Updated 10 months ago
- data cleaning and curation for unstructured text☆328Updated last year
- awesome synthetic (text) datasets☆302Updated 3 months ago
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆289Updated 7 months ago
- Let's build better datasets, together!☆262Updated 10 months ago
- Fast & more realistic evaluation of chat language models. Includes leaderboard.☆189Updated last year
- code for training & evaluating Contextual Document Embedding models☆198Updated 5 months ago
- an implementation of Self-Extend, to expand the context window via grouped attention☆118Updated last year
- Notebooks for training universal 0-shot classifiers on many different tasks☆136Updated 9 months ago
- ☆415Updated last year
- Convert all of libgen to high quality markdown☆252Updated last year
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆256Updated 2 years ago
- ☆88Updated last year
- Just a bunch of benchmark logs for different LLMs☆118Updated last year
- A bagel, with everything.☆324Updated last year
- 📝 Reference-Free automatic summarization evaluation with potential hallucination detection☆102Updated last year
- ☆159Updated 10 months ago
- A compact LLM pretrained in 9 days by using high quality data☆330Updated 6 months ago
- ☆210Updated 3 months ago