superlinear-ai / wtpsplit-liteLinks
✂️ Sentence segmentation with wtpsplit's state-of-the-art Segment any Text (SaT) models
☆13Updated last week
Alternatives and similar repositories for wtpsplit-lite
Users that are interested in wtpsplit-lite are comparing it to the libraries listed below
Sorting:
- German Language Understanding Evaluation Benchmark @NAACL24☆10Updated this week
- Source code and data for Like a Good Nearest Neighbor☆29Updated 5 months ago
- Temporary remove unused tokens during training to save ram and speed.☆23Updated last week
- Code for SaGe subword tokenizer (EACL 2023)☆25Updated 6 months ago
- My NER Experiments with ModernBERT☆21Updated last month
- ☆10Updated 8 months ago
- A Python library aimed at dissecting and augmenting NER training data.☆58Updated 2 years ago
- NLP with Rust for Python 🦀🐍☆62Updated last month
- Starbucks: Improved Training for 2D Matryoshka Embeddings☆21Updated 4 months ago
- GLADIS: A General and Large Acronym Disambiguation Benchmark (EACL 23)☆16Updated last year
- Library for fast text representation and classification.☆30Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆58Updated last month
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆27Updated last year
- ☆27Updated 4 months ago
- Minimum Bayes Risk Decoding for Hugging Face Transformers☆58Updated last year
- Multilingual Entity Linking model by BELA model☆12Updated last year
- Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"☆19Updated last year
- ☆12Updated 6 months ago
- ☆42Updated 2 years ago
- ☆13Updated 3 years ago
- INCOME: An Easy Repository for Training and Evaluation of Index Compression Methods in Dense Retrieval. Includes BPR and JPQ.☆24Updated last year
- [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"☆32Updated 3 weeks ago
- Simple-to-use scoring function for arbitrarily tokenized texts.☆41Updated 4 months ago
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆25Updated 3 months ago
- Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.☆82Updated 9 months ago
- SeqScore: Scoring for named entity recognition and other sequence labeling tasks☆23Updated 3 months ago
- ☆61Updated last week
- MergeBench: A Benchmark for Merging Domain-Specialized LLMs☆14Updated last month
- ☆21Updated 3 years ago
- Learning BPE embeddings by first learning a segmentation model and then training word2vec☆19Updated 2 years ago