SumanthRH / tokenizationLinks
A comprehensive deep dive into the world of tokens
☆225Updated last year
Alternatives and similar repositories for tokenization
Users that are interested in tokenization are comparing it to the libraries listed below
Sorting:
- Website for hosting the Open Foundation Models Cheat Sheet.☆267Updated 2 months ago
- Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free☆232Updated 9 months ago
- experiments with inference on llama☆104Updated last year
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated last year
- A set of scripts and notebooks on LLM finetunning and dataset creation☆110Updated 10 months ago
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆197Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆268Updated last year
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆284Updated 5 months ago
- An introduction to LLM Sampling☆79Updated 7 months ago
- data cleaning and curation for unstructured text☆328Updated last year
- Generate textbook-quality synthetic LLM pretraining data☆502Updated last year
- awesome synthetic (text) datasets☆291Updated 3 weeks ago
- ☆93Updated last year
- A bagel, with everything.☆323Updated last year
- Let's build better datasets, together!☆260Updated 7 months ago
- code for training & evaluating Contextual Document Embedding models☆196Updated 2 months ago
- A puzzle to learn about prompting☆132Updated 2 years ago
- ☆87Updated last year
- 📝 Reference-Free automatic summarization evaluation with potential hallucination detection☆101Updated last year
- Just a bunch of benchmark logs for different LLMs☆119Updated last year
- ☆124Updated 9 months ago
- LoRA and DoRA from Scratch Implementations☆207Updated last year
- Convert all of libgen to high quality markdown☆253Updated last year
- ☆40Updated last year
- batched loras☆344Updated last year
- Fast bare-bones BPE for modern tokenizer training☆160Updated last month
- Collection of autoregressive model implementation☆86Updated 3 months ago
- Highly commented implementations of Transformers in PyTorch☆136Updated 2 years ago
- Fast & more realistic evaluation of chat language models. Includes leaderboard.☆187Updated last year
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆256Updated last year