tianlinxu312 / Everything-about-LLMsLinks
A work in progress. Trying to write about all interesting or necessary pieces in the current development of LLMs and generative AI. Gradually adding more topics.
☆199Updated 2 years ago
Alternatives and similar repositories for Everything-about-LLMs
Users that are interested in Everything-about-LLMs are comparing it to the libraries listed below
Sorting:
- LoRA and DoRA from Scratch Implementations☆215Updated last year
- Official implementation of TransNormerLLM: A Faster and Better LLM☆248Updated last year
- Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models☆137Updated last year
- Implementation of CALM from the paper "LLM Augmented LLMs: Expanding Capabilities through Composition", out of Google Deepmind☆178Updated last year
- This code repository contains the code used for my "Optimizing Memory Usage for Training LLMs and Vision Transformers in PyTorch" blog po…☆92Updated 2 years ago
- Implementation of the Llama architecture with RLHF + Q-learning☆168Updated 10 months ago
- [ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”☆125Updated 10 months ago
- Inference code for Mistral and Mixtral hacked up into original Llama implementation☆371Updated last year
- Project 2 (Building Large Language Models) for Stanford CS324: Understanding and Developing Large Language Models (Winter 2022)☆105Updated 2 years ago
- Scaling Data-Constrained Language Models☆342Updated 5 months ago
- Pre-training code for Amber 7B LLM☆169Updated last year
- a Fine-tuned LLaMA that is Good at Arithmetic Tasks☆178Updated 2 years ago
- A comprehensive deep dive into the world of tokens☆227Updated last year
- Implementation of paper Data Engineering for Scaling Language Models to 128K Context☆478Updated last year
- Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)☆205Updated last year
- An extension of the nanoGPT repository for training small MOE models.☆215Updated 8 months ago
- Implementation of DoRA☆307Updated last year
- ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward exp…☆225Updated 2 months ago
- ☆272Updated 2 years ago
- Data preparation code for Amber 7B LLM☆93Updated last year
- NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day☆258Updated 2 years ago
- A minimal implementation of LLaVA-style VLM with interleaved image & text & video processing ability.☆96Updated 11 months ago
- Low-bit optimizers for PyTorch☆132Updated 2 years ago
- The Truth Is In There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction☆389Updated last year
- minimal GRPO implementation from scratch☆100Updated 8 months ago
- Code for paper titled "Towards the Law of Capacity Gap in Distilling Language Models"☆102Updated last year
- ☆205Updated last week
- PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention…☆294Updated last year
- This repository contains an implementation of the LLaMA 2 (Large Language Model Meta AI) model, a Generative Pretrained Transformer (GPT)…☆74Updated 2 years ago
- Micro Llama is a small Llama based model with 300M parameters trained from scratch with $500 budget☆162Updated 3 months ago