pchizhov / picky_bpeLinks
BPE modification that implements removing of the intermediate tokens during tokenizer training.
☆25Updated 6 months ago
Alternatives and similar repositories for picky_bpe
Users that are interested in picky_bpe are comparing it to the libraries listed below
Sorting:
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆55Updated 2 weeks ago
- Code for SaGe subword tokenizer (EACL 2023)☆25Updated 6 months ago
- Repository containing the SPIN experiments on the DIBT 10k ranked prompts☆24Updated last year
- ☆25Updated last year
- A framework for pitting LLMs against each other in an evolving library of games ⚔☆32Updated last month
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆57Updated 9 months ago
- ☆12Updated 5 months ago
- ☆49Updated 6 months ago
- This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…☆31Updated last year
- Pre-train Static Word Embeddings☆70Updated this week
- ☆55Updated 3 weeks ago
- PyTorch implementation for MRL☆18Updated last year
- ReBase: Training Task Experts through Retrieval Based Distillation☆29Updated 3 months ago
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…☆26Updated last year
- Embedding Recycling for Language models☆38Updated last year
- ☆45Updated 4 months ago
- 🕸 GlotCC Dataset and Pipline -- NeurIPS 2024☆19Updated last month
- ☆13Updated 2 weeks ago
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Updated last year
- QLoRA for Masked Language Modeling☆22Updated last year
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆32Updated last year
- ☆40Updated last year
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…☆48Updated last year
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆25Updated 2 months ago
- Aioli: A unified optimization framework for language model data mixing☆25Updated 4 months ago
- Official implementation of "BERTs are Generative In-Context Learners"☆28Updated 2 months ago
- The source code of our work "Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models" [AISTATS …☆59Updated 7 months ago
- ☆44Updated 6 months ago
- ☆68Updated 9 months ago
- Download, parse, and filter data from Phil Papers. Data-ready for The-Pile.☆16Updated last year