KorAP / Tokenizer-EvaluationLinks
Benchmark scripts for comparing different tokenizers and sentence segmenters of German
☆12Updated 2 years ago
Alternatives and similar repositories for Tokenizer-Evaluation
Users that are interested in Tokenizer-Evaluation are comparing it to the libraries listed below
Sorting:
- NLP with Rust for Python 🦀🐍☆71Updated 8 months ago
- Pre-train Static Word Embeddings☆94Updated 5 months ago
- Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …☆69Updated last month
- Efficiently computing & storing token n-grams from large corpora☆26Updated last year
- A polite and user-friendly downloader for Common Crawl data☆67Updated 5 months ago
- A sentence segmentation library with wide language support optimized for speed and utility.☆86Updated 3 weeks ago
- Contextualized per-token embeddings☆34Updated 9 months ago
- 🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)☆17Updated last year
- Extracts plain text, language identification and more metadata from WARC records☆23Updated 4 months ago
- Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…☆68Updated 2 months ago
- lossily compress representation vectors using product quantization☆59Updated 3 months ago
- PyLate efficient inference engine☆71Updated last month
- This repository contains an easy and intuitive approach to use SetFit in combination with spaCy.☆81Updated 2 years ago
- [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.☆111Updated last year
- Libraries, Archives and Museums (LAM)☆88Updated 3 years ago
- Faster, modernized fork of the language identification tool langid.py☆60Updated last year
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆230Updated last month
- Completion After Prompt Probability. Make your LLM make a choice☆82Updated last year
- Code for SaGe subword tokenizer (EACL 2023)☆27Updated last year
- A Python module for retrieving script types of writing systems including alphabets, abjads, abugidas, syllabaries, logographs, featurals …☆15Updated last year
- ☆50Updated 3 months ago
- Tree-based indexes for neural-search☆31Updated last year
- GraphER: A Structure-aware Text-to-Graph Model for Entity and Relation Extraction☆84Updated last year
- The Batched API provides a flexible and efficient way to process multiple requests in a batch, with a primary focus on dynamic batching o…☆155Updated 6 months ago
- Notebooks for training universal 0-shot classifiers on many different tasks☆140Updated last year
- Library for fast text representation and classification.☆31Updated 2 years ago
- 🔢 Work with static vector models☆36Updated 9 months ago
- Small python package to measure OCR quality and other related metrics.☆26Updated last year
- This is a new backend implementation of the ANNIS linguistic search and visualization system.☆18Updated 3 weeks ago
- A robust web archive analytics toolkit☆129Updated 3 months ago