SumanthRH / tokenizationLinks
A comprehensive deep dive into the world of tokens
☆223Updated 11 months ago
Alternatives and similar repositories for tokenization
Users that are interested in tokenization are comparing it to the libraries listed below
Sorting:
- Manage scalable open LLM inference endpoints in Slurm clusters☆258Updated 10 months ago
- Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free☆230Updated 7 months ago
- A bagel, with everything.☆320Updated last year
- ☆517Updated 6 months ago
- data cleaning and curation for unstructured text☆327Updated 9 months ago
- Website for hosting the Open Foundation Models Cheat Sheet.☆267Updated 3 weeks ago
- ☆152Updated 6 months ago
- A set of scripts and notebooks on LLM finetunning and dataset creation☆111Updated 8 months ago
- Just a bunch of benchmark logs for different LLMs☆119Updated 10 months ago
- Minimal example scripts of the Hugging Face Trainer, focused on staying under 150 lines☆196Updated last year
- experiments with inference on llama☆104Updated 11 months ago
- ☆92Updated last year
- Toolkit for attaching, training, saving and loading of new heads for transformer models☆279Updated 2 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆81Updated last year
- Generate textbook-quality synthetic LLM pretraining data☆498Updated last year
- Let's build better datasets, together!☆258Updated 5 months ago
- inference code for mixtral-8x7b-32kseqlen☆99Updated last year
- Fast & more realistic evaluation of chat language models. Includes leaderboard.☆187Updated last year
- ☆40Updated last year
- Highly commented implementations of Transformers in PyTorch☆136Updated last year
- Code for the paper "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples"☆302Updated last year
- An introduction to LLM Sampling☆78Updated 5 months ago
- Fast bare-bones BPE for modern tokenizer training☆157Updated 2 months ago
- code for training & evaluating Contextual Document Embedding models☆191Updated 3 weeks ago
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆254Updated last year
- ☆210Updated 11 months ago
- Convert all of libgen to high quality markdown☆250Updated last year
- Notebooks for training universal 0-shot classifiers on many different tasks☆126Updated 5 months ago
- Simple Transformer in Jax☆137Updated 11 months ago
- A puzzle to learn about prompting☆127Updated 2 years ago