pchizhov / picky_bpe
BPE modification that implements removing of the intermediate tokens during tokenizer training.
☆25Updated 5 months ago
Alternatives and similar repositories for picky_bpe
Users that are interested in picky_bpe are comparing it to the libraries listed below
Sorting:
- Code for SaGe subword tokenizer (EACL 2023)☆24Updated 5 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆47Updated last week
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment☆57Updated 8 months ago
- ☆56Updated last week
- ☆48Updated 6 months ago
- Repository containing the SPIN experiments on the DIBT 10k ranked prompts☆24Updated last year
- ReBase: Training Task Experts through Retrieval Based Distillation☆29Updated 3 months ago
- Code for the examples presented in the talk "Training a Llama in your backyard: fine-tuning very large models on consumer hardware" given…☆14Updated last year
- ☆39Updated last week
- lossily compress representation vectors using product quantization☆52Updated 3 weeks ago
- ☆43Updated 3 months ago
- This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…☆31Updated last year
- ☆45Updated 3 months ago
- 🕸 GlotCC Dataset and Pipline -- NeurIPS 2024☆18Updated last month
- Pre-train Static Word Embeddings☆60Updated last month
- Plug-and-play Search Interfaces with Pyserini and Hugging Face☆31Updated last year
- Embedding Recycling for Language models☆38Updated last year
- QLoRA for Masked Language Modeling☆22Updated last year
- Repository for the Q-Filters method (https://arxiv.org/pdf/2503.02812)☆30Updated 2 months ago
- ☆56Updated last week
- Codes and files for the paper Are Emergent Abilities in Large Language Models just In-Context Learning☆33Updated 4 months ago
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…☆34Updated last year
- Code for Zero-Shot Tokenizer Transfer☆127Updated 4 months ago
- ☆63Updated 7 months ago
- Python library to use Pleias-RAG models☆46Updated 2 weeks ago
- Download, parse, and filter data from Phil Papers. Data-ready for The-Pile.☆16Updated last year
- Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"☆24Updated 2 months ago
- A fast, local, and secure approach for training LLMs for coding tasks using GRPO with WebAssembly and interpreter feedback.☆22Updated last month
- ☆22Updated 3 months ago
- ☆15Updated last month