本项目从零开始构建并优化了一个千万参数级别的大规模预训练语言模型，涵盖预训练、有监督微调（SFT）和R1推理蒸馏三个阶段。项目采用自定义Transformer架构（包括RMSNorm、分组注意力、多Query机制、SwiGLU激活和RoPE位置编码），实现高效的长文本处理和自回归生成。同时，开发了基于BBPE编码的分词器，针对中文进行了优化，构建了轻量级词表。预训练阶段通过混合精度训练、梯度累积和Cosine Annealing学习率调度，实现全流程训练；微调阶段则利用开源SFT数据集，通过特殊损失掩码优化指令遵循；R1推理蒸馏阶段采用Deepseek-R1数据及损失加权策略，赋予模型逐步推理（慢思考）的能力，显著提升复杂任务的回答质量。
☆21Mar 10, 2025Updated last year
Alternatives and similar repositories for WeakWater-30M
Users that are interested in WeakWater-30M are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Image captioning with a locally stored Large Language Model (LLM)☆16Updated this week
- 北京邮电大学网络工程嵌入式系统实验报告☆12Jan 7, 2021Updated 5 years ago
- 在千问最新的多模态image-text模型Qwen3-VL-4B-Instruct 进行多种lora微调对比效果，通过langchain+RAG+多智能体(Multi-Agent)进行部署☆32Dec 14, 2025Updated 3 months ago
- 使用Decoder-only的Transformer进行时序预测，包含SwiGLU和RoPE(Rotary Positional Embedding)，Time series prediction using Decoder-only Transformer, Includ…☆16Jan 25, 2024Updated 2 years ago
- BUPT神经网络与深度学习课设☆10Dec 29, 2023Updated 2 years ago
- Simple, predictable pricing with DigitalOcean hosting • AdAlways know what you'll pay with monthly caps and flat pricing. Enterprise-grade infrastructure trusted by 600k+ customers.
- learn with ipython notebook and stepby step. 用 ipython notebook 来学各种算法， 力争做到小步学习☆11Mar 6, 2019Updated 7 years ago
- 🏆 SUCCESS-GS: Survey of Compactness and Compression for Efficient Static and Dynamic Gaussian Splatting☆26Feb 4, 2026Updated last month
- 本项目对Deepseek-R1-Distill-Qwen-7B进行心理咨询CoT 数据的LoRA微调，以进一步提升Deepseek-R1-Distill-Qwen-7B在心理咨询领域的慢思考能力。☆12Mar 11, 2025Updated last year
- [AAAI' 26]SparseSurf: Sparse-View 3D Gaussian Splatting for Surface Reconstruction☆26Nov 19, 2025Updated 4 months ago
- [ICCV25] MGSR: 2D/3D Mutual-boosted Gaussian Splatting for High-fidelity Surface Reconstruction under Various Light Conditions☆21Feb 19, 2026Updated last month
- Intelligent Robot Interaction Lab Assignment, SAI of BUPT. 北邮智能机器人交互实验仿真作业☆14Mar 17, 2025Updated last year
- bupt nlp第二次作业：分别基于SVD分解以及基于SGNS两种方法构建汉语子词向量并进行评测☆10May 16, 2023Updated 2 years ago
- 大模型API企业网关，公司内部API管理，分发聚和系统，支持将多种大模型转换成统一的OpenAI兼容接口，尤其对国内开源模型deepseek,qwen，kimi，glm提供特别支持 可供个人或者企业内部大模型API统一管理和渠道分发使用(key管理与二次分发)，长期更新，支…☆41Sep 12, 2025Updated 6 months ago
- 一个基于transformers的自定义命名实体识别模型示例☆17Jul 4, 2021Updated 4 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- [ICCV 2025] Distilling Parallel Gradients for Fast ODE Solvers of Diffusion Models☆35Mar 20, 2026Updated last week
- ☆13Feb 25, 2022Updated 4 years ago
- 2021级BUPT深度学习与神经网络课程设计源代码☆15Jan 21, 2024Updated 2 years ago
- 基于LLaVA1.6微调的Xray识别的多模态大模型☆10Oct 22, 2024Updated last year
- BUPT Software Engineering Project☆18Aug 20, 2018Updated 7 years ago
- My implementation of 《"GrabCut" - Interactive Foreground Extraction using Iterated Graph Cuts》☆16Aug 7, 2018Updated 7 years ago
- 最基本最小白的自然语言处理入门读物，基于deepseek-r1，涵盖了传统NLP和现代大模型☆23Jan 16, 2026Updated 2 months ago
- Implementations of a Mixture-of-Experts (MoE) architecture designed for research on large language models (LLMs) and scalable neural netw…☆63Apr 8, 2025Updated 11 months ago
- How to use ProtocolBuffer☆10Apr 4, 2017Updated 8 years ago
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Third Party RDP Wrapper Library☆24Mar 19, 2026Updated last week
- ☆19Jun 14, 2024Updated last year
- BUPT智能计算系统☆21Jan 2, 2024Updated 2 years ago
- A collection of metric learning papers.☆20Apr 24, 2023Updated 2 years ago
- 前后端分离的答题系统，前端对应pc的网页，和对应微信的小程序端，前期主要对微信小程序进行完善。前端网页设计使用vue技术，后端使用SpringBoot等相关技术☆15Aug 18, 2020Updated 5 years ago
- An android app which converts text/voice input to American Sign Language(ASL)☆14Sep 8, 2016Updated 9 years ago
- ☆45Oct 5, 2025Updated 5 months ago
- ☆17Apr 16, 2021Updated 4 years ago
- 完成了《实战Google深度学习框架》里的内容☆20Oct 6, 2018Updated 7 years ago
- DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- The MongoDB Database☆22Dec 7, 2016Updated 9 years ago
- ☆21Mar 11, 2025Updated last year
- A project that can generate ancient poems based on pictures, including CLIP, T5, GPT2 models☆22Feb 16, 2025Updated last year
- Official repository for the paper "EAP-GS: Efficient Augmentation of Pointcloud for 3D Gaussian Splatting in Few-shot Scene Reconstructio…☆32Jun 15, 2025Updated 9 months ago
- Qt Webkit and ExtJs☆19Nov 20, 2013Updated 12 years ago
- 通过lora对deepseek小模型进行微调☆23Nov 15, 2024Updated last year
- flutter lib android e ios salvar e abrir documents tipo pdf, ppt e excel ...etc☆12Jun 18, 2024Updated last year