EleutherAI / tokengramsLinks
Efficiently computing & storing token n-grams from large corpora
ā26Updated 10 months ago
Alternatives and similar repositories for tokengrams
Users that are interested in tokengrams are comparing it to the libraries listed below
Sorting:
- NLP with Rust for Python š¦šā64Updated 3 months ago
- This repository contains code for cleaning your training data of benchmark data to help combat data snooping.ā25Updated 2 years ago
- Code for SaGe subword tokenizer (EACL 2023)ā26Updated 8 months ago
- Plug-and-play Search Interfaces with Pyserini and Hugging Faceā32Updated 2 years ago
- minimal pytorch implementation of bm25 (with sparse tensors)ā104Updated last year
- Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/ā¦ā27Updated last year
- Supercharge huggingface transformers with model parallelism.ā77Updated last month
- new optimizerā20Updated last year
- Understanding how features learned by neural networks evolve throughout trainingā36Updated 10 months ago
- ā64Updated last month
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.ā43Updated last month
- ā49Updated 6 months ago
- ā41Updated 3 months ago
- ā62Updated 2 weeks ago
- Training code for Sparse Autoencoders on Embedding modelsā38Updated 5 months ago
- Experiments for efforts to train a new and improved t5ā76Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.ā63Updated 2 weeks ago
- Pre-train Static Word Embeddingsā85Updated 2 months ago
- [COLM '24] Source-Aware Training Enables Knowledge Attribution in Language Modelsā19Updated 4 months ago
- ā56Updated 3 months ago
- ā38Updated last year
- URL downloader supporting checkpointing and continuous checksumming.ā19Updated last year
- Small python package to measure OCR quality and other related metrics.ā25Updated last year
- QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning Pā¦ā34Updated 2 years ago
- Training hybrid models for dummies.ā25Updated 7 months ago
- š¤ Disaggregators: Curated data labelers for in-depth analysis.ā66Updated 2 years ago
- See https://github.com/cuda-mode/triton-index/ instead!ā11Updated last year
- ā57Updated 11 months ago
- An unofficial implementation of the Infini-gram model proposed by Liu et al. (2024)ā33Updated last year
- PyTorch implementation for MRLā19Updated last year