KorAP / Tokenizer-EvaluationLinks
Benchmark scripts for comparing different tokenizers and sentence segmenters of German
☆12Updated 2 years ago
Alternatives and similar repositories for Tokenizer-Evaluation
Users that are interested in Tokenizer-Evaluation are comparing it to the libraries listed below
Sorting:
- Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …☆55Updated last week
- NLP with Rust for Python 🦀🐍☆65Updated 4 months ago
- A polite and user-friendly downloader for Common Crawl data☆57Updated last month
- Efficiently computing & storing token n-grams from large corpora☆26Updated 11 months ago
- Contextualized per-token embeddings☆29Updated 4 months ago
- Pre-train Static Word Embeddings☆85Updated 2 weeks ago
- Extracts plain text, language identification and more metadata from WARC records☆23Updated 2 weeks ago
- This is a new backend implementation of the ANNIS linguistic search and visualization system.☆18Updated 3 weeks ago
- Fast Text Classification with Compressors dictionary☆150Updated 2 years ago
- Layout Analysis Dataset with Segmonto (LADaS)☆21Updated 2 months ago
- ☆42Updated 2 weeks ago
- 🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)☆17Updated last year
- Library for fast text representation and classification.☆31Updated last year
- Next-generation Punkt sentence boundary detection with zero dependencies☆17Updated last month
- Highly concurrent and fast content processing for Mighty Inference Server☆10Updated 2 years ago
- Code for SaGe subword tokenizer (EACL 2023)☆26Updated 9 months ago
- A general purpose processing framework for corpora of scientific documents☆65Updated last week
- A client library for LAION's effort to filter CommonCrawl with CLIP, building a large scale image-text dataset.☆32Updated 2 years ago
- A Python module for retrieving script types of writing systems including alphabets, abjads, abugidas, syllabaries, logographs, featurals …☆15Updated last year
- Libraries, Archives and Museums (LAM)☆85Updated 2 years ago
- This repository contains an easy and intuitive approach to use SetFit in combination with spaCy.☆80Updated 2 years ago
- Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…☆67Updated 2 months ago
- Like picoGPT but for BERT.☆50Updated 2 years ago
- Python library to use Pleias-RAG models☆62Updated 4 months ago
- Inference engine for GLiNER models, in Rust☆67Updated 2 months ago
- Small python package to measure OCR quality and other related metrics.☆25Updated last year
- 🤗 Disaggregators: Curated data labelers for in-depth analysis.☆67Updated 2 years ago
- Training code for Sparse Autoencoders on Embedding models☆38Updated 6 months ago
- ☆157Updated 2 years ago
- ☆60Updated last year