Alternatives and similar repositories for infini-gram

Users that are interested in infini-gram are comparing it to the libraries listed below

• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆63Updated 2 weeks ago
• bminixhofer / tokenkit
  A toolkit implementing advanced methods to transfer models and model knowledge across tokenizers.
  ☆43Updated last month
• jxmorris12 / bm25_pt
  minimal pytorch implementation of bm25 (with sparse tensors)
  ☆104Updated last year
• allenai / bff
  ☆39Updated last year
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆85Updated 2 months ago
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆49Updated last year
• HazyResearch / aioli
  Aioli: A unified optimization framework for language model data mixing
  ☆27Updated 7 months ago
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated 2 years ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆26Updated 8 months ago
• ielab / Starbucks
  Starbucks: Improved Training for 2D Matryoshka Embeddings
  ☆21Updated last month
• mungg / FABLES
  ☆57Updated 11 months ago
• hadasah / btm
  ☆75Updated last year
• princeton-nlp / LitSearch
  [EMNLP 2024] A Retrieval Benchmark for Scientific Literature Search
  ☆94Updated 8 months ago
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆77Updated last month
• castorini / umbrela
  ☆53Updated last month
• google-research-datasets / QAmeleon
  QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…
  ☆34Updated 2 years ago
• srush / LLM-Talk
  ☆51Updated last year
• chandar-lab / NeoBERT
  ☆80Updated 2 months ago
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆26Updated 2 years ago
• ibm-granite / granite-embedding-models
  ☆33Updated 2 months ago
• Pleias / Pleias-RAG-Library
  Python library to use Pleias-RAG models
  ☆61Updated 3 months ago
• schen149 / sub-sentence-encoder
  The official code repo for "Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations".
  ☆83Updated last year
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆59Updated 2 weeks ago
• facebookresearch / lss_eval
  This is a new metric that can be used to evaluate faithfulness of text generated by LLMs. The work behind this repository can be found he…
  ☆31Updated 2 years ago
• ZeroSumEval / ZeroSumEval
  A framework for pitting LLMs against each other in an evolving library of games ⚔
  ☆33Updated 4 months ago
• allenai / infinigram-api
  ☆77Updated last week
• ContextualAI / CLAIR_and_APO
  Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  ☆60Updated 11 months ago
• allenai / catwalk
  This project studies the performance and robustness of language models and task-adaptation methods.
  ☆151Updated last year
• thomasnormal / fewshot
  ☆28Updated 2 months ago
• allenai / peS2o
  Pretraining Efficiently on S2ORC!
  ☆165Updated 10 months ago