owos / flexitokensLinks
FlexiTokens
☆18Updated 2 months ago
Alternatives and similar repositories for flexitokens
Users that are interested in flexitokens are comparing it to the libraries listed below
Sorting:
- State-of-the-art paired encoder and decoder models (17M-1B params)☆53Updated 4 months ago
- Code for SaGe subword tokenizer (EACL 2023)☆27Updated last year
- ☆15Updated 2 months ago
- Starbucks: Improved Training for 2D Matryoshka Embeddings☆22Updated 5 months ago
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆25Updated last year
- Official implementation of "GPT or BERT: why not both?"☆63Updated 4 months ago
- ☆58Updated 3 weeks ago
- Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders☆18Updated 6 months ago
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆49Updated 2 years ago
- Code repository for the paper "MrT5: Dynamic Token Merging for Efficient Byte-level Language Models."☆51Updated 2 months ago
- ☆13Updated last year
- ☆55Updated 10 months ago
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆31Updated 2 months ago
- ☆14Updated last year
- https://footprints.baulab.info☆17Updated last year
- Official code for the NeurIPS25 paper "RAT: Bridging RNN Efficiencyand Attention Accuracy in Language Modeling" (https://arxiv.org/abs/25…☆21Updated this week
- Improving Text Embedding of Language Models Using Contrastive Fine-tuning☆66Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆67Updated 2 months ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆60Updated last year
- ☆23Updated 10 months ago
- MEXMA: Token-level objectives improve sentence representations☆42Updated 11 months ago
- Embedding Recycling for Language models☆38Updated 2 years ago
- Label shift estimation for transfer difficulty with Familiarity.☆10Updated 10 months ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated 2 years ago
- 🕸 GlotCC Dataset and Pipline -- NeurIPS 2024☆20Updated 8 months ago
- Aioli: A unified optimization framework for language model data mixing☆31Updated 10 months ago
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.☆57Updated 5 months ago
- PathPiece tokenizer☆13Updated last year
- Official code release for "SuperBPE: Space Travel for Language Models"☆76Updated 3 weeks ago
- Fast search index for SPLADE sparse retrieval models implemented in Python using Numpy and Numba☆31Updated last month