Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram
☆22Oct 21, 2026Updated last week
Alternatives and similar repositories for script_tok
Users that are interested in script_tok are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- ☆12Dec 24, 2025Updated 10 months ago
- PathPiece tokenizer☆14Nov 10, 2024Updated last year
- Demo project for YOLO implementation with Rust and WASM☆11Mar 29, 2024Updated 2 years ago
- Code for Retrofitting Large Language Models with Dynamic Tokenization.☆14Jul 22, 2025Updated last year
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆27Nov 25, 2024Updated last year
- Managed Kubernetes at scale on DigitalOcean • AdDigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
- Claude's Tokenizer, Offline, Kinda☆97Sep 25, 2026Updated last month
- Code for SaGe subword tokenizer (EACL 2023)☆28Nov 30, 2024Updated last year
- ☆12Oct 10, 2026Updated 2 weeks ago
- ☆11Jan 27, 2025Updated last year
- Master thesis: Exploring bias in German NLG (GPT-3 & GerPT-2). Applies regard classification and bias mitigation triggers.☆16Sep 25, 2024Updated 2 years ago
- [ACL 2025] 🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment☆11Apr 6, 2025Updated last year
- The training codes of Jasper-Token-Compression-600M☆22Nov 19, 2025Updated 11 months ago
- UI themes for Sabaki.☆12Nov 5, 2018Updated 7 years ago
- Get plain text from Wikipedia pages☆11Jul 2, 2025Updated last year
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"☆22Feb 14, 2024Updated 2 years ago
- [NeurIPS 2025] MergeBench: A Benchmark for Merging Domain-Specialized LLMs☆54Aug 28, 2026Updated 2 months ago
- This repository provides code for the paper "Memory-Efficient Fine-Tuning of Transformers via Token Selection" (EMNLP 2024).☆14Feb 11, 2025Updated last year
- Hyperparam local dataset viewer☆29Sep 21, 2026Updated last month
- Ukrainian ELECTRA model☆12Mar 11, 2023Updated 3 years ago
- decontamination☆40Oct 17, 2026Updated last week
- SCT: An Efficient Self-Supervised Cross-View Training For Sentence Embedding (TACL)☆16Jul 27, 2024Updated 2 years ago
- Efficient and simple data mixing with on-policy proxies!☆22May 18, 2026Updated 5 months ago
- ☆17Aug 22, 2026Updated 2 months ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- ☆11Mar 17, 2026Updated 7 months ago
- 🌍 A simple script for taking automated screenshots from a Leaflet map☆16Mar 29, 2018Updated 8 years ago
- Code base for the EMNLP 2021 Findings paper: Cartography Active Learning☆14Jun 3, 2025Updated last year
- A Multilingual Keyboard Layout-Based Typo Generator☆18Nov 23, 2025Updated 11 months ago
- TensorFlow implementation of "Generating Sentences from a Continuous Space"☆11Sep 16, 2019Updated 7 years ago
- ☆109Jun 2, 2025Updated last year
- suffix array construction and searching algorithms for in-memory binary data.☆13Sep 10, 2022Updated 4 years ago
- From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks☆15Feb 23, 2023Updated 3 years ago
- Simple-to-use scoring function for arbitrarily tokenized texts.☆52Oct 18, 2026Updated last week
- Wordpress hosting with auto-scaling - Free Trial Offer • AdFully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
- codebase release for EMNLP2023 paper publication☆19Sep 18, 2025Updated last year
- Measuring the Signal to Noise Ratio in Language Model Evaluation☆32Aug 19, 2025Updated last year
- This is the repository for MorphScore, a tokenizer evaluation framework for morphological alignment.☆19Jul 10, 2025Updated last year
- Raw-to-End Name Entity Recognition in Social Media☆16Oct 16, 2019Updated 7 years ago
- Multi-Word Probabilistic based supertokenizer☆15May 15, 2025Updated last year
- FlexiTokens☆23Dec 27, 2025Updated 10 months ago
- A curated list of tokenizer libraries for blazing-fast NLP processing.☆14Apr 10, 2025Updated last year