本项目从零开始构建并优化了一个千万参数级别的大规模预训练语言模型，涵盖预训练、有监督微调（SFT）和R1推理蒸馏三个阶段。项目采用自定义Transformer架构（包括RMSNorm、分组注意力、多Query机制、SwiGLU激活和RoPE位置编码），实现高效的长文本处理和自回归生成。同时，开发了基于BBPE编码的分词器，针对中文进行了优化，构建了轻量级词表。预训练阶段通过混合精度训练、梯度累积和Cosine Annealing学习率调度，实现全流程训练；微调阶段则利用开源SFT数据集，通过特殊损失掩码优化指令遵循；R1推理蒸馏阶段采用Deepseek-R1数据及损失加权策略，赋予模型逐步推理（慢思考）的能力，显著提升复杂任务的回答质量。
☆22Mar 10, 2025Updated last year
Alternatives and similar repositories for WeakWater-30M
Users that are interested in WeakWater-30M are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Image captioning with a locally stored Large Language Model (LLM)☆15Updated this week
- 北京邮电大学网络工程嵌入式系统实验报告☆12Jan 7, 2021Updated 5 years ago
- 使用Decoder-only的Transformer进行时序预测，包含SwiGLU和RoPE(Rotary Positional Embedding)，Time series prediction using Decoder-only Transformer, Includ…☆16Jan 25, 2024Updated 2 years ago
- BUPT神经网络与深度学习课设☆10Dec 29, 2023Updated 2 years ago
- learn with ipython notebook and stepby step. 用 ipython notebook 来学各种算法， 力争做到小步学习☆11Mar 6, 2019Updated 7 years ago
- Bare Metal GPUs on DigitalOcean Gradient AI • AdPurpose-built for serious AI teams training foundational models, running large-scale inference, and pushing the boundaries of what's possible.
- 🏆 SUCCESS-GS: Survey of Compactness and Compression for Efficient Static and Dynamic Gaussian Splatting☆33Apr 10, 2026Updated last month
- 本项目对Deepseek-R1-Distill-Qwen-7B进行心理咨询CoT数据的LoRA微调，以进一步提升Deepseek-R1-Distill-Qwen-7B在心理咨询领域的慢思考能力。☆12Mar 11, 2025Updated last year
- Intelligent Robot Interaction Lab Assignment, SAI of BUPT. 北邮智能机器人交互实验仿真作业☆15Mar 17, 2025Updated last year
- [ICCV25] MGSR: 2D/3D Mutual-boosted Gaussian Splatting for High-fidelity Surface Reconstruction under Various Light Conditions☆20Feb 19, 2026Updated 3 months ago
- 大模型API企业网关，公司内部API管理，分发聚和系统，支持将多种大模型转换成统一的OpenAI兼容接口，尤其对国内开源模型deepseek,qwen，kimi，glm提供特别支持 可供个人或者企业内部大模型API统一管理和渠道分发使用(key管理与二次分发)，长期更新，支…☆40Sep 12, 2025Updated 8 months ago
- 一个基于transformers的自定义命名实体识别模型示例☆17Jul 4, 2021Updated 4 years ago
- 一个基于 模型上下文协议/MCP 构建的智能医学文献分析工具。它旨在帮助科研人员、医学从业者和学生快速检索 PubMed 数据库，并利用大型语言模型 (LLM) 的能力对文献摘要进行智能分析和总结☆10May 18, 2025Updated last year
- 企业级多Agent知识管理系统：4个AI Agent协作完成文档解析→知识抽取→智能问答→增量更新，Python/Java/Go三语言实现☆71Apr 5, 2026Updated last month
- 个人学习的医疗大模型微调项目☆34Dec 3, 2025Updated 5 months ago
- Open source password manager - Proton Pass • AdSecurely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
- ☆13Feb 25, 2022Updated 4 years ago
- 2021级BUPT深度学习与神经网络课程设计源代码☆15Jan 21, 2024Updated 2 years ago
- 在千问最新的多模态image-text模型Qwen3-VL-4B-Instruct 进行多种lora微调对比效果，通过langchain+RAG+多智能体(Multi-Agent)进行部署☆48Dec 14, 2025Updated 5 months ago
- TPAMI + ICCV25☆36May 6, 2026Updated 2 weeks ago
- BUPT Software Engineering Project☆18Aug 20, 2018Updated 7 years ago
- [AAAI' 26]SparseSurf: Sparse-View 3D Gaussian Splatting for Surface Reconstruction☆27May 11, 2026Updated 2 weeks ago
- 基于LLaVA1.6微调的Xray识别的多模态大模型☆10Oct 22, 2024Updated last year
- My implementation of 《"GrabCut" - Interactive Foreground Extraction using Iterated Graph Cuts》☆16Aug 7, 2018Updated 7 years ago
- How to use ProtocolBuffer☆10Apr 4, 2017Updated 9 years ago
- Managed Database hosting by DigitalOcean • AdPostgreSQL, MySQL, MongoDB, Kafka, Valkey, and OpenSearch available. Automatically scale up storage and focus on building your apps.
- Third Party RDP Wrapper Library☆25Updated this week
- Implementations of a Mixture-of-Experts (MoE) architecture designed for research on large language models (LLMs) and scalable neural netw…☆71Apr 8, 2025Updated last year
- ☆19Jun 14, 2024Updated last year
- BUPT智能计算系统☆21Jan 2, 2024Updated 2 years ago
- A collection of metric learning papers.☆21Apr 24, 2023Updated 3 years ago
- 前后端分离的答题系统，前端对应pc的网页，和对应微信的小程序端，前期主要对微信小程序进行完善。前端网页设计使用vue技术，后端使用SpringBoot等相关技术☆15Aug 18, 2020Updated 5 years ago
- 最基本最小白的自然语言处理入门读物，基于deepseek-r1，涵盖了传统NLP和现代大模型☆27Jan 16, 2026Updated 4 months ago
- A LangGraph-powered multi-agent deep research system featuring task planning, human-in-the-loop review, multi-source retrieval, evidence-…☆50May 11, 2026Updated 2 weeks ago
- An android app which converts text/voice input to American Sign Language(ASL)☆14Sep 8, 2016Updated 9 years ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- ☆17Apr 16, 2021Updated 5 years ago
- 完成了《实战Google深度学习框架》里的内容☆20Oct 6, 2018Updated 7 years ago
- The MongoDB Database☆22Dec 7, 2016Updated 9 years ago
- ☆22Mar 11, 2025Updated last year
- A project that can generate ancient poems based on pictures, including CLIP, T5, GPT2 models☆21Feb 16, 2025Updated last year
- Official repository for the paper "EAP-GS: Efficient Augmentation of Pointcloud for 3D Gaussian Splatting in Few-shot Scene Reconstructio…☆31Jun 15, 2025Updated 11 months ago
- Qt Webkit and ExtJs☆19Nov 20, 2013Updated 12 years ago