Benchmark scripts for comparing different tokenizers and sentence segmenters of German
☆12Feb 27, 2023Updated 3 years ago
Alternatives and similar repositories for Tokenizer-Evaluation
Users that are interested in Tokenizer-Evaluation are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.
Sorting:
- Translation of query languages to serialized KoralQuery protocol☆14Mar 9, 2026Updated 2 weeks ago
- A Corpus Data Retrieval Index using Lucene for Look-Ups☆20Updated this week
- ☆13Nov 28, 2025Updated 3 months ago
- Paris multilayer transport network☆11Sep 13, 2021Updated 4 years ago
- Searching in-memory corpus with Corpus Query Language (CQL)☆19Dec 2, 2024Updated last year
- Coquery is a free corpus query tool for linguists, lexicographers, translators, and anybody who wishes to search and analyse a text corpu…☆19Jan 7, 2026Updated 2 months ago
- ☆11Feb 13, 2026Updated last month
- Paytm Payments MCP Server☆24Aug 27, 2025Updated 6 months ago
- Multi Tier Annotation Search☆12May 13, 2024Updated last year
- Course materials for the MVA course "algorithms for speech and language processing"☆12Mar 29, 2023Updated 2 years ago
- natural language processing on german texts☆16Mar 20, 2018Updated 8 years ago
- Layout Analysis Dataset with Segmonto (LADaS)☆24Jul 12, 2025Updated 8 months ago
- ☆16Aug 22, 2023Updated 2 years ago
- This is a new backend implementation of the ANNIS linguistic search and visualization system.☆18Jan 13, 2026Updated 2 months ago
- Chapel-based Optimization☆14Oct 26, 2025Updated 4 months ago
- Collection de romans français du dix-huitième siècle (1751-1800) / Collection of Eighteenth-Century French Novels (1751-1800)☆23Apr 23, 2024Updated last year
- ☆20Jan 9, 2026Updated 2 months ago
- Korpuslinguistik war noch nie so einfach...☆25Feb 18, 2026Updated last month
- shoco is a compressor for small text strings. [Not maintained].☆10Sep 4, 2019Updated 6 years ago
- code and data used to build a training dataset for dragnet models☆10Nov 29, 2020Updated 5 years ago
- Hexatomic is an extensible software for deep multi-layer annotation of linguistic corpora☆17Nov 27, 2024Updated last year
- Camel Morph’s goal is to build large open-source morphological models for Arabic and its dialects across many genres and domains.☆15Dec 8, 2024Updated last year
- A fully-fledge PyTorch package for Morphological Analysis, tailored to morphologically rich and historical languages.☆24Oct 27, 2023Updated 2 years ago
- Basis of FragDenStaat.de's „Koalitionstracker“☆15Jul 14, 2025Updated 8 months ago
- ☆10Dec 21, 2024Updated last year
- Small string compression using smaz compression algorithm. Fast, because it's in C. Supports Python 3+☆13Oct 18, 2025Updated 5 months ago
- 🕸 GlotWeb: Web Indexing for Minority Languages (WWW 2026)☆17Feb 27, 2026Updated 3 weeks ago
- The Wikinflection Corpus, from the paper "Wikinflection Corpus: A (Better) Multilingual, Morpheme-Annotated Inflectional Corpus" (Metheni…☆12Dec 15, 2023Updated 2 years ago
- Benson turns a list of URLs into mp3s of the contents of each web page - take control over your reading backlog!☆16Oct 30, 2024Updated last year
- Automated Safety Testing of Large Language Models☆18Jan 31, 2025Updated last year
- Get started writing a Chrome extension in TypeScript!☆18Aug 3, 2023Updated 2 years ago
- A reddit bot that finds original publish dates on linked articles.☆10Nov 30, 2024Updated last year
- Static Huffman coding☆10Apr 3, 2017Updated 8 years ago
- SMOR (Stuttgart Morphology) with alternative lemmatization component☆13Aug 10, 2023Updated 2 years ago
- Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki☆28Jul 31, 2024Updated last year
- A set of utilities to turn Dataclasses into useful configuration managers.☆11Mar 27, 2024Updated last year
- A tiny graph database engine written in C☆10May 9, 2014Updated 11 years ago
- Produces a serialized hardware report of the physical infrastructure for automation☆26Jan 27, 2026Updated last month
- Supervised Stylometry☆27Mar 4, 2026Updated 2 weeks ago