superlinear-ai / wtpsplit-liteLinks
✂️ Sentence segmentation with wtpsplit's state-of-the-art Segment any Text (SaT) models
☆17Updated last month
Alternatives and similar repositories for wtpsplit-lite
Users that are interested in wtpsplit-lite are comparing it to the libraries listed below
Sorting:
- Temporary remove unused tokens during training to save ram and speed.☆24Updated last month
- German Language Understanding Evaluation Benchmark @NAACL24☆12Updated 2 weeks ago
- Code for SaGe subword tokenizer (EACL 2023)☆25Updated 8 months ago
- Library for fast text representation and classification.☆31Updated last year
- GLADIS: A General and Large Acronym Disambiguation Benchmark (EACL 23)☆17Updated last year
- Starbucks: Improved Training for 2D Matryoshka Embeddings☆21Updated last month
- INCOME: An Easy Repository for Training and Evaluation of Index Compression Methods in Dense Retrieval. Includes BPR and JPQ.☆24Updated last year
- My NER Experiments with ModernBERT and Ettin☆22Updated 3 weeks ago
- Minimum Bayes Risk Decoding for Hugging Face Transformers☆58Updated last year
- NLP with Rust for Python 🦀🐍☆64Updated 2 months ago
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆27Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆63Updated 2 months ago
- ☆10Updated 10 months ago
- Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"☆20Updated last year
- [EMNLP'23] Official Code for "FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models"☆32Updated 2 months ago
- State-of-the-art paired encoder and decoder models (17M-1B params)☆38Updated last week
- ☆51Updated 6 months ago
- Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.☆82Updated 11 months ago
- Model implementation for the contextual embeddings project☆35Updated 2 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- ☆79Updated 2 months ago
- Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning☆30Updated 2 years ago
- Source code and data for Like a Good Nearest Neighbor☆30Updated 7 months ago
- One-stop shop for running and fine-tuning transformer-based language models for retrieval☆57Updated this week
- SeqScore: Scoring for named entity recognition and other sequence labeling tasks☆23Updated 4 months ago
- LTG-Bert☆33Updated last year
- Pre-train Static Word Embeddings☆85Updated 2 months ago
- ☆27Updated 5 months ago
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆27Updated 5 months ago
- Crispy reranking models by Mixedbread☆34Updated 3 weeks ago