LAGoM-NLP / transtokenizerLinks
☆52Updated 8 months ago
Alternatives and similar repositories for transtokenizer
Users that are interested in transtokenizer are comparing it to the libraries listed below
Sorting:
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆65Updated this week
- ☆58Updated 4 months ago
- Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).☆80Updated last year
- Code for Zero-Shot Tokenizer Transfer☆137Updated 8 months ago
- A massively multilingual modern encoder language model☆91Updated last week
- Pre-train Static Word Embeddings☆85Updated 3 weeks ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆59Updated last year
- State-of-the-art paired encoder and decoder models (17M-1B params)☆50Updated last month
- Code repository for the paper "MrT5: Dynamic Token Merging for Efficient Byte-level Language Models."☆51Updated last week
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.☆46Updated 2 months ago
- ☆48Updated last year
- Improving Text Embedding of Language Models Using Contrastive Fine-tuning☆64Updated last year
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆49Updated last year
- ☆49Updated 7 months ago
- Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…☆67Updated 3 months ago
- BPE modification that implements removing of the intermediate tokens during tokenizer training.☆24Updated 10 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- The first dense retrieval model that can be prompted like an LM☆89Updated 4 months ago
- ☆57Updated last year
- Notebooks for training universal 0-shot classifiers on many different tasks☆137Updated 9 months ago
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆30Updated 2 weeks ago
- ☆83Updated 4 months ago
- MEXMA: Token-level objectives improve sentence representations☆41Updated 8 months ago
- Official implementation of "GPT or BERT: why not both?"☆59Updated 2 months ago
- Datasets collection and preprocessings framework for NLP extreme multitask learning☆186Updated 2 months ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated 2 years ago
- Using open source LLMs to build synthetic datasets for direct preference optimization☆66Updated last year
- Python library to use Pleias-RAG models☆62Updated 5 months ago
- Maya: An Instruction Finetuned Multilingual Multimodal Model using Aya☆116Updated last month
- Library to facilitate pruning of LLMs based on context☆32Updated last year