r-three / common-pileLinks
Code for collecting, processing, and preparing datasets for the Common Pile
☆227Updated last week
Alternatives and similar repositories for common-pile
Users that are interested in common-pile are comparing it to the libraries listed below
Sorting:
- Python library to use Pleias-RAG models☆62Updated 4 months ago
- ☆81Updated 2 weeks ago
- ☆190Updated 2 months ago
- Code for SaGe subword tokenizer (EACL 2023)☆26Updated 9 months ago
- Small python package to measure OCR quality and other related metrics.☆25Updated last year
- ☆249Updated 5 months ago
- ☆58Updated 4 months ago
- An introduction to LLM Sampling☆79Updated 9 months ago
- ☆57Updated 11 months ago
- minimal pytorch implementation of bm25 (with sparse tensors)☆104Updated last year
- ☆52Updated 10 months ago
- ☆68Updated last month
- ☆84Updated 3 months ago
- code for training & evaluating Contextual Document Embedding models☆197Updated 4 months ago
- ☆67Updated last year
- Trully flash implementation of DeBERTa disentangled attention mechanism.☆63Updated 2 weeks ago
- ☆49Updated 7 months ago
- A massively multilingual modern encoder language model☆80Updated 2 weeks ago
- A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.☆46Updated 2 months ago
- ☆135Updated last month
- Pre-train Static Word Embeddings☆85Updated 2 weeks ago
- lossily compress representation vectors using product quantization☆59Updated 5 months ago
- Using open source LLMs to build synthetic datasets for direct preference optimization☆65Updated last year
- Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…☆67Updated 2 months ago
- Code and data to support "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4"☆69Updated 2 years ago
- Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).☆80Updated last year
- An easy-to-understand framework for LLM samplers that rewind and revise generated tokens☆146Updated 7 months ago
- Code for Zero-Shot Tokenizer Transfer☆137Updated 8 months ago
- Datamodels for hugging face tokenizers☆71Updated this week
- ☆142Updated 2 weeks ago