MeLeLBGU / tokenizers_intrinsic_benchmarkLinks
Code for the paper "Greed is All You Need: An Evaluation of Tokenizer Inference Methods"
☆12Updated last year
Alternatives and similar repositories for tokenizers_intrinsic_benchmark
Users that are interested in tokenizers_intrinsic_benchmark are comparing it to the libraries listed below
Sorting:
- ☆231Updated 4 years ago
- [Kauf & Ivanova, ACL 2023] A Better Way to Do Masked Language Model Scoring☆10Updated 2 years ago
- A simple library for querying the URIEL typological database.☆93Updated last year
- Automated Semantic Analysis of Discourse Markers☆10Updated 3 years ago
- Diagnostic tests for linguistic capacities in language models☆65Updated 3 years ago
- An initiative to collect and distribute resources for co-reference resolution in a unified standard.☆25Updated last year
- Utility for behavioral and representational analyses of Language Models☆173Updated last week
- The Benchmark of Linguistic Minimal Pairs☆159Updated 3 years ago
- A neural word aligner based on multilingual BERT☆362Updated 3 years ago
- ☆33Updated 3 months ago
- Obtain Word Alignments using Pretrained Language Models (e.g., mBERT)☆385Updated 2 years ago
- Repository for DISRPT2023 shared task☆17Updated last year
- A tool that locates, downloads, and extracts machine translation corpora☆159Updated 3 months ago
- Various utility scripts useful for natural language processing, machine translation, etc.☆50Updated 3 years ago
- Efficient Low-Memory Aligner☆146Updated 11 months ago
- MT Evaluation in Many Languages via Zero-Shot Paraphrasing☆102Updated last year
- This repository houses the IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of >25k semiautomatically generated se…☆19Updated 4 years ago
- This repository provides details and links to the ACL anthology corpus/collection including .bib, .pdf and grobid extractions of the pdfs☆187Updated 2 years ago
- PyTorch source code of NAACL 2021 paper "Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Tran…☆18Updated 3 years ago
- Easier Automatic Sentence Simplification Evaluation☆165Updated 2 years ago
- A tool for holistic analysis of language generations systems☆471Updated 3 months ago
- Discourse Probing of Pretrained Language Models. In Proceedings of NAACL 2021.☆10Updated 3 years ago
- Appraise code used as part of WMT21 human evaluation campaign☆29Updated last week
- ☆15Updated 5 years ago
- This is a repository with the data and code for the ACL 2019 paper "When a Good Translation is Wrong in Context: ..." and the EMNLP 2019 …☆98Updated 5 years ago
- Evaluating recurrent neural networks on predicting subject-verb agreement dependencies☆63Updated 2 years ago
- Appraise evaluation system for manual evaluation of machine translation output☆77Updated 4 years ago
- Lexical Substitution Framework☆46Updated 2 years ago
- ☆29Updated last year
- STREUSLE: a corpus with comprehensive lexical semantic annotation (multiword expressions, supersenses)☆68Updated last month