Convert your language model into a byte-level one!
☆14Feb 26, 2026Updated 8 months ago
Alternatives and similar repositories for byte-sampler
Users that are interested in byte-sampler are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- When is multilinguality a curse? Language modeling for 250 high- and low-resource languages (EMNLP 2024).☆11Jan 30, 2025Updated last year
- Repository for "Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages"☆15Oct 4, 2024Updated 2 years ago
- Ukrainian ELECTRA model☆12Mar 11, 2023Updated 3 years ago
- ☆26Dec 12, 2025Updated 10 months ago
- A library for language transfer methods and algorithms.☆16Feb 6, 2026Updated 8 months ago
- GPUs on demand by Runpod - Special Offer Available • AdRun AI, ML, and HPC workloads on powerful cloud GPUs—without limits or wasted spend. Deploy GPUs in under a minute and pay by the second.
- From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks☆15Feb 23, 2023Updated 3 years ago
- suffix array construction and searching algorithms for in-memory binary data.☆13Sep 10, 2022Updated 4 years ago
- Code for Retrofitting Large Language Models with Dynamic Tokenization.☆14Jul 22, 2025Updated last year
- Python module to remove wiki markup text.☆10Jan 15, 2016Updated 10 years ago
- Code for ACL 2022 paper "Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation"☆29Apr 2, 2022Updated 4 years ago
- ☆42May 26, 2026Updated 5 months ago
- High-performance backend for language model probabilistic programs☆17Oct 14, 2026Updated 3 weeks ago
- CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation☆14Aug 19, 2025Updated last year
- A text-to-SQL prototype on the northwind sqlite dataset☆13Sep 24, 2024Updated 2 years ago
- 1-Click AI Models by DigitalOcean Gradient • AdDeploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
- CTC beam search☆12Oct 26, 2016Updated 10 years ago
- Fast and simple server to host state-of-the-art retrieval models☆15Jun 4, 2026Updated 5 months ago
- Code for the paper "BPE stays on SCRIPT", "Which Pieces Does Unigram Tokenization Really Need?" and MinGram☆22Updated this week
- [TACL 2021] Code and data for the framework in "Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-La…☆115Mar 24, 2022Updated 4 years ago
- [ACL 2024] 💬 MaskLID: Code-Switching Language Identification through Iterative Masking☆13Updated this week
- Master thesis: Exploring bias in German NLG (GPT-3 & GerPT-2). Applies regard classification and bias mitigation triggers.☆16Sep 25, 2024Updated 2 years ago
- ☆13Oct 17, 2022Updated 4 years ago
- Gorilla: An API store for LLMs☆10Jun 28, 2024Updated 2 years ago
- A research framework for studying how multi-agent LLM teams complete software tasks given different team structures and sizes.☆16Apr 16, 2026Updated 6 months ago
- AI Agents on DigitalOcean Gradient AI Platform • AdBuild production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
- A collection of English tweets annotated in Universal Dependencies.☆39Oct 20, 2021Updated 5 years ago
- Rust binding to crfsuite☆25Updated this week
- Scaling Sparse Fine-Tuning to Large Language Models☆20Jan 31, 2024Updated 2 years ago
- Utilities to gather software metrics from tools (SONAR, etc) and store them into ElasticSearch for later display using Kibana.☆11Dec 31, 2017Updated 8 years ago
- Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"☆22Feb 14, 2024Updated 2 years ago
- Decontextualizing Snippets from Scientific Documents with Large Language Models☆13Oct 10, 2023Updated 3 years ago
- The official implementation of the paper "Self-Updatable Large Language Models by Integrating Context into Model Parameters"☆16May 18, 2025Updated last year
- Qwen3.5-0.8B writes a Bend program; the same model type-checks and runs it inside its own weights. No compiler, no interpreter, no tool c…☆24Updated this week
- Crawler based on a modified browser to detect online tracking.☆11Jul 19, 2023Updated 3 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- Lightweight piece tokenization library☆12Aug 24, 2026Updated 2 months ago
- ☆12Updated this week
- An implementation of JSON Lines for Rust☆16Apr 11, 2021Updated 5 years ago
- CNER: Concept and Named Entity Recognition☆13Jul 15, 2024Updated 2 years ago
- FastText Pytorch version☆12Apr 22, 2022Updated 4 years ago
- Residual Quantization Autoencoder, used for interpreting LLMs☆14Jan 1, 2025Updated last year
- A collection of utilities for handling IPA phones.☆27Sep 24, 2023Updated 3 years ago