hpcaitech / PaLM-colossalai
Scalable PaLM implementation of PyTorch
☆190Updated 2 years ago
Alternatives and similar repositories for PaLM-colossalai:
Users that are interested in PaLM-colossalai are comparing it to the libraries listed below
- Performance benchmarking with ColossalAI☆39Updated 2 years ago
- Official repository for DistFlashAttn: Distributed Memory-efficient Attention for Long-context LLMs Training☆209Updated 8 months ago
- ☆104Updated last year
- Code used for sourcing and cleaning the BigScience ROOTS corpus☆309Updated 2 years ago
- ☆117Updated last year
- DSIR large-scale data selection framework for language model training☆246Updated last year
- train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism☆218Updated last year
- GPTQ inference Triton kernel☆300Updated last year
- Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.☆95Updated last year
- Ongoing research training transformer language models at scale, including: BERT & GPT-2☆69Updated last year
- Examples of training models with hybrid parallelism using ColossalAI☆339Updated 2 years ago
- ☆97Updated last year
- Fast Inference Solutions for BLOOM☆561Updated 6 months ago
- A unified tokenization tool for Images, Chinese and English.☆152Updated 2 years ago
- A LLaMA1/LLaMA12 Megatron implement.☆28Updated last year
- A collection of models built with ColossalAI☆32Updated 2 years ago
- Running BERT without Padding☆471Updated 3 years ago
- Open Instruction Generalist is an assistant trained on massive synthetic instructions to perform many millions of tasks☆208Updated last year
- REST: Retrieval-Based Speculative Decoding, NAACL 2024☆199Updated 4 months ago
- Official repository for LongChat and LongEval☆519Updated 11 months ago
- This is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/…☆95Updated last year
- Large Scale Distributed Model Training strategy with Colossal AI and Lightning AI☆57Updated last year
- Simple implementation of Speculative Sampling in NumPy for GPT-2.☆93Updated last year
- 📑 Dive into Big Model Training☆111Updated 2 years ago
- Scaling Data-Constrained Language Models☆334Updated 7 months ago
- Code and models for BERT on STILTs☆53Updated 2 years ago
- A (somewhat) minimal library for finetuning language models with PPO on human feedback.☆85Updated 2 years ago
- The aim of this repository is to utilize LLaMA to reproduce and enhance the Stanford Alpaca☆97Updated 2 years ago
- Experiments on speculative sampling with Llama models☆125Updated last year
- ☆411Updated last year