Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and transformer models. Header-only and zero-dependency.
☆16Aug 7, 2025Updated last year
Alternatives and similar repositories for Modern-Text-Tokenizer
Users that are interested in Modern-Text-Tokenizer are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- BASIC reborn for the AI era: APL-style arrays, SDL3 games, native LLVM compile, Win32 forms, local LLMs, and a built-in MCP server your A…☆17Updated this week
- Port of Andrej Karpathy's llama2.c to Kotlin.☆21Sep 18, 2023Updated 2 years ago
- DVI for HSTX☆24Mar 23, 2025Updated last year
- ☆37Updated this week
- Zeta implementation of a reusable and plug in and play feedforward from the paper "Exponentially Faster Language Modeling"☆16Nov 11, 2024Updated last year
- Deploy open-source AI quickly and easily - Special Bonus Offer • AdRunpod Hub is built for open source. One-click deployment and autoscaling endpoints without provisioning your own infrastructure.
- Coq & Haskell code for Calculating Correct Compilers II☆12Feb 22, 2022Updated 4 years ago
- Lambda layer to enable using famous NLTK python package with AWS lambda☆10Mar 21, 2024Updated 2 years ago
- Sandboxed code execution for LLMs, powered by Nix☆16Jul 23, 2026Updated last month
- Simple VPN client in Docker container☆13Apr 11, 2022Updated 4 years ago
- DVI (HDMI) and VGA display driver for Pico2 RP2350, using HSTX peripheral☆49Aug 17, 2026Updated last week
- 读取多层级目录下的pdf文件，通常是爬虫爬下来的，将其中摘要和正文抽取出来。可以快速抽取想要的文本内容。☆16May 12, 2022Updated 4 years ago
- 在您的机器上本地离线运行 AI 模型☆11May 8, 2025Updated last year
- The implementation of Text Classification with Negative Supervision (ACL, 2020)☆10Oct 8, 2020Updated 5 years ago
- 使用Sentencepiece对中文语料进行分词☆13Nov 30, 2023Updated 2 years ago
- Deploy to Railway using AI coding agents - Free Credits Offer • AdUse Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
- UniRobot is an embodied intelligent software framework that integrates the robot brain (data, models, model training) with the robot body…☆18Nov 8, 2025Updated 9 months ago
- ☆16Apr 13, 2024Updated 2 years ago
- ☆17Jan 31, 2025Updated last year
- Yet another frontend for LLM, written using .NET and WinUI 3☆11Sep 14, 2025Updated 11 months ago
- Optimizing database queries with array programming☆20Sep 21, 2020Updated 5 years ago
- AI-powered Software Requirements Analysis (SRA) system that evaluates requirement quality, detects ambiguities, and suggests structured i…☆22Aug 21, 2026Updated last week
- Code for "Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation" (Findings of ACL 2024)☆16Jul 4, 2024Updated 2 years ago
- Interactive Brokers Native Python API☆14Dec 26, 2022Updated 3 years ago
- A desktop GUI for Flux 1.1 Pro built using DelphiFMX For Python☆11Oct 5, 2024Updated last year
- End-to-end encrypted cloud storage - Proton Drive • AdSpecial offer: 40% Off Yearly / 80% Off First Month. Protect your most important files, photos, and documents from prying eyes.
- ICDE 2025 Paper, Grounding Natural Language to SQL Translation with Data-Based Self-Explanations☆17May 24, 2025Updated last year
- Ubuntu 18.04 Docker container with TightVNC server☆29Mar 27, 2020Updated 6 years ago
- A2UI Client that communicates with A2A (Agent-to-Agent) servers using the A2A protocol with optional A2UI extensions for rich interactive…☆16Jan 25, 2026Updated 7 months ago
- Simple agent framework using Ollama tool calling☆10Aug 27, 2024Updated 2 years ago
- GFPGAN face reconstruction with ncnn on a bare Raspberry Pi☆14Jan 4, 2023Updated 3 years ago
- 一个即插即用的llm增强器，项目初衷是通过工程化架构让小参数模型获得能媲美大模型的能力。☆19May 23, 2026Updated 3 months ago
- Lightweight API Specification for Intelligent Systems☆15Feb 16, 2026Updated 6 months ago
- Windows desktop application for Stable Diffusion☆13Jun 26, 2023Updated 3 years ago
- Protocol for transparent collections of files.☆12May 17, 2026Updated 3 months ago
- Deploy on Railway without the complexity - Free Credits Offer • AdConnect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
- Valori is a Deterministic Memory OS that sits between intelligence (LLMs) and reality (devices, products, decisions).☆18Updated this week
- Haskell bindings to llama.cpp☆12Dec 22, 2023Updated 2 years ago
- LCM Drawing app☆12Dec 1, 2023Updated 2 years ago
- Yet Another (LLM) Web UI, made with Gemini☆12Dec 25, 2024Updated last year
- Identify which embedding model produced a vector using digit-level tokenization and a tiny transformer☆23Mar 7, 2026Updated 5 months ago
- Run commands concurrently with rust and cargo☆10Jan 20, 2023Updated 3 years ago
- ojjson is a library designed to facilitate JSON interactions with Ollama, a large language api (LLM). It leverages the power of Zod for s…☆12Nov 7, 2024Updated last year