SumanthRH / tokenizationLinks
A comprehensive deep dive into the world of tokens
☆226Updated last year
Alternatives and similar repositories for tokenization
Users that are interested in tokenization are comparing it to the libraries listed below
Sorting:
- Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free☆232Updated 9 months ago
- Website for hosting the Open Foundation Models Cheat Sheet.☆267Updated 3 months ago
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆197Updated last year
- A puzzle to learn about prompting☆132Updated 2 years ago
- experiments with inference on llama☆104Updated last year
- A set of scripts and notebooks on LLM finetunning and dataset creation☆110Updated 10 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆82Updated last year
- Highly commented implementations of Transformers in PyTorch☆137Updated 2 years ago
- ☆93Updated last year
- data cleaning and curation for unstructured text☆328Updated last year
- Generate textbook-quality synthetic LLM pretraining data☆504Updated last year
- Let's build better datasets, together!☆261Updated 8 months ago
- Convert all of libgen to high quality markdown☆254Updated last year
- An introduction to LLM Sampling☆79Updated 8 months ago
- Fast bare-bones BPE for modern tokenizer training☆164Updated 2 months ago
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆285Updated 5 months ago
- awesome synthetic (text) datasets☆293Updated last month
- Project 2 (Building Large Language Models) for Stanford CS324: Understanding and Developing Large Language Models (Winter 2022)☆105Updated 2 years ago
- batched loras☆345Updated last year
- LLM Workshop by Sourab Mangrulkar☆392Updated last year
- ☆155Updated 8 months ago
- code for training & evaluating Contextual Document Embedding models☆197Updated 3 months ago
- ☆88Updated last year
- Manage scalable open LLM inference endpoints in Slurm clusters☆270Updated last year
- Just a bunch of benchmark logs for different LLMs☆120Updated last year
- Simple Transformer in Jax☆139Updated last year
- A bagel, with everything.☆324Updated last year
- ☆210Updated 2 months ago
- ☆124Updated 9 months ago
- ☆416Updated last year