cisnlp / GlotCCLinks
πΈ GlotCC Dataset and Pipline -- NeurIPS 2024
β20Updated 4 months ago
Alternatives and similar repositories for GlotCC
Users that are interested in GlotCC are comparing it to the libraries listed below
Sorting:
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.β43Updated last month
- π Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentβ11Updated 4 months ago
- Using short models to classify long textsβ21Updated 2 years ago
- MEXMA: Token-level objectives improve sentence representationsβ41Updated 7 months ago
- β22Updated 7 months ago
- BLOOM+1: Adapting BLOOM model to support a new unseen languageβ73Updated last year
- SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectsβ22Updated 7 months ago
- β14Updated 3 months ago
- Code for Zero-Shot Tokenizer Transferβ135Updated 7 months ago
- β51Updated 7 months ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Faceβ32Updated 2 years ago
- A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.β59Updated last year
- BPE modification that implements removing of the intermediate tokens during tokenizer training.β24Updated 9 months ago
- Repository containing the SPIN experiments on the DIBT 10k ranked promptsβ24Updated last year
- Minimum Bayes Risk Decoding for Hugging Face Transformersβ58Updated last year
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/β¦β27Updated last year
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning Pβ¦β34Updated 2 years ago
- Official implementation of "GPT or BERT: why not both?"β57Updated last month
- Embedding Recycling for Language modelsβ39Updated 2 years ago
- β57Updated 11 months ago
- ππ€ A collection of templates for Hugging Face Spacesβ35Updated last year
- Using open source LLMs to build synthetic datasets for direct preference optimizationβ65Updated last year
- Code for SaGe subword tokenizer (EACL 2023)β26Updated 8 months ago
- β13Updated 8 months ago
- Trully flash implementation of DeBERTa disentangled attention mechanism.β63Updated 2 weeks ago
- Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentβ60Updated 11 months ago
- Implementation of "SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages" paper, accepted to Eβ¦β23Updated 2 years ago
- β51Updated 2 years ago
- Repo for training MLMs, CLMs, or T5-type models on the OLM pretraining data, but it should work with any hugging face text dataset.β93Updated 2 years ago
- SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 laβ¦β49Updated last year