WangHuiNEU / Transformer_Knowlegde
从底层机理了解Transformer
☆27Updated 3 years ago
Alternatives and similar repositories for Transformer_Knowlegde:
Users that are interested in Transformer_Knowlegde are comparing it to the libraries listed below
- 基于Gated Attention Unit的Transformer模型(尝鲜版)☆97Updated 2 years ago
- A light-weight script for maintaining a LOT of machine learning experiments.☆91Updated 2 years ago
- FLASHQuad_pytorch☆67Updated 3 years ago
- dpo算法实现☆33Updated 9 months ago
- an implementation of transformer, bert, gpt, and diffusion models for learning purposes☆152Updated 5 months ago
- ☆186Updated 5 months ago
- Lion and Adam optimization comparison☆60Updated 2 years ago
- The Roadmap for LLMs☆84Updated last year
- ☆52Updated last year
- The official repo of INF-34B models trained by INF Technology.☆34Updated 8 months ago
- A Tight-fisted Optimizer☆47Updated 2 years ago
- pytorch单精度、半精度、混合精度、单卡、多卡(DP / DDP)、FSDP、DeepSpeed模型训练代码,并对比不同方法的训练速度以及GPU内存的使用☆96Updated last year
- ☆189Updated last year
- DeepSpeed教程 & 示例注释 & 学习笔记 (大模型高效训练)☆156Updated last year
- RoFormer V1 & V2 pytorch☆491Updated 2 years ago
- 更纯粹、更高压缩率的Tokenizer☆473Updated 4 months ago
- Code for a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models☆62Updated last month
- Codes for "Learning Sparse Sharing Architectures for Multiple Tasks"☆93Updated 4 years ago
- The pure and clear PyTorch Distributed Training Framework.☆276Updated last year
- A collection of phenomenons observed during the scaling of big foundation models, which may be developed into consensus, principles, or l…☆279Updated last year
- 实现了Transformer中的几种位置编码方案☆41Updated 3 years ago
- ☆106Updated 4 months ago
- Rectified Rotary Position Embeddings☆362Updated 10 months ago
- Paper List for In-context Learning 🌷☆180Updated last year
- A paper list of pre-trained language models (PLMs).☆80Updated 3 years ago
- [ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408☆196Updated last year
- NTK scaled version of ALiBi position encoding in Transformer.☆67Updated last year
- A paper list about diffusion models for natural language processing.☆182Updated last year
- LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training☆400Updated 2 weeks ago
- [ICLR 2024]EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling(https://arxiv.org/abs/2310.04691)☆121Updated last year