Due to the huge vocaburary size (151,936) of Qwen models, the Embedding and LM Head weights are excessively heavy. Therefore, this project provides a Tokenizer vocabulary shearing solution for Qwen and Qwen-VL.
☆34Jan 6, 2026Updated 2 months ago
Alternatives and similar repositories for Qwen-Tokenizer-Pruner
Users that are interested in Qwen-Tokenizer-Pruner are comparing it to the libraries listed below
Sorting:
- 本项目提供了基于910B的huggingface LLM模型的Tensor Parallel(TP)部署教程，同时也可以作为一份极简的TP学习代码。☆32Jan 6, 2026Updated 2 months ago
- Codebase for Instruction Following without Instruction Tuning☆36Sep 24, 2024Updated last year
- My Implementation of Q-Sparse: All Large Language Models can be Fully Sparsely-Activated☆33Aug 14, 2024Updated last year
- Conic10K: A large-scale dataset for closed-vocabulary math problem understanding. Accepted to EMNLP2023 Findings.☆31Dec 6, 2023Updated 2 years ago
- InSales e-commerce platform API bindings☆14Jul 13, 2024Updated last year
- Official implementation of "Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation" (CVPR 202…☆40May 26, 2025Updated 9 months ago
- ☆44Nov 1, 2025Updated 4 months ago
- ☆11Feb 19, 2022Updated 4 years ago
- ☆54May 19, 2025Updated 9 months ago
- ☆29Dec 20, 2025Updated 2 months ago
- Implementation for the CVPR2019 paper "Graphical Contrastive Losses for Scene Graph Parsing"☆12Nov 11, 2019Updated 6 years ago
- Implementation of a fast semantic chunker in C++, installable in python 3.7+ projects.☆22Sep 20, 2025Updated 5 months ago
- A tool for generating synthetic data.☆18Dec 19, 2025Updated 2 months ago
- Nanos klib for NVIDIA GPUs☆14Mar 25, 2025Updated 11 months ago
- ☆13Nov 5, 2024Updated last year
- Russian phonetical transcription☆11Nov 19, 2025Updated 3 months ago
- Simple repository for training small reasoning models☆49Feb 17, 2026Updated 2 weeks ago
- CLIPCleaner: Cleaning Noisy Labels with CLIP (ACM MM2024)☆13Apr 28, 2025Updated 10 months ago
- Code for the paper "Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching" (COLING 2025)☆19Jan 3, 2026Updated 2 months ago
- ☆10Jul 18, 2023Updated 2 years ago
- The PyTorch implementation of DSM (EMNLP 2022).☆10Mar 26, 2024Updated last year
- JAX implementation of GPTQ quantization algorithm☆10Jul 19, 2023Updated 2 years ago
- Code showing how to use a model based on the ML model base class.☆10Sep 30, 2022Updated 3 years ago
- Pdf Query chat-bot using Gemini AI and Llma Index☆10Dec 24, 2023Updated 2 years ago
- Piper based VoiceDock TTS implementation☆11Aug 12, 2023Updated 2 years ago
- Các thí nghiệm liên quan tới LLMs cho tiếng Việt (insprised by Physics of LLMs Series)☆11Oct 21, 2024Updated last year
- [ICML 2025 Oral] This is the official repository of the paper "What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensi…☆21Jun 12, 2025Updated 8 months ago
- Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions☆23Feb 11, 2026Updated 3 weeks ago
- JAX notebook showing how to LoRA + GPTQ arbitrary models☆10Aug 8, 2023Updated 2 years ago
- ☆10Dec 26, 2023Updated 2 years ago
- ☆14Oct 21, 2024Updated last year
- This repository contains the dataset of the paper ARGUS: Context-Based Detection of Stealthy IoT Infiltration Attacks☆12Apr 28, 2023Updated 2 years ago
- Python package for compressing floating-point PyTorch tensors☆13Jul 22, 2024Updated last year
- This repository collects awesome representative papers and resources for "From Pre-training to Post-training: A Survey on Time Series Fou…☆31Feb 1, 2026Updated last month
- 大型中文道德句数据集CMOS☆10Apr 11, 2022Updated 3 years ago
- [CIKM 2022] Towards Automated Over-Sampling for Imbalanced Classification☆10Mar 20, 2023Updated 2 years ago
- Tokenizer for Text to Speech (TTS) models☆13Jan 16, 2025Updated last year
- A hackable library for running and fine-tuning modern transformer models on commodity and alternative GPUs, powered by tinygrad.☆28Feb 10, 2026Updated 3 weeks ago
- Package, experimentation results, and other artifacts for the serverless computing performance modeling paper.☆10Jun 22, 2022Updated 3 years ago