Alternatives and similar repositories for transtokenizer:

Users that are interested in transtokenizer are comparing it to the libraries listed below

• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆53Updated this week
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆25Updated 4 months ago
• Hannibal046 / nanoColBERT
  Simple replication of [ColBERT-v1](https://arxiv.org/abs/2004.12832).
  ☆80Updated last year
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆57Updated 8 months ago
• chandar-lab / NeoBERT
  ☆44Updated last month
• Aleph-Alpha / trigrams
  ☆54Updated 7 months ago
• YutongWang1216 / DocMTAgent
  Code and data releases for the paper -- DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory
  ☆39Updated 2 months ago
• unicamp-dl / InRanker
  ☆47Updated last year
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆48Updated last year
• orionw / promptriever
  The first dense retrieval model that can be prompted like an LM
  ☆68Updated 6 months ago
• jfkback / hypencoder-paper
  Official Repository for "Hypencoder: Hypernetworks for Information Retrieval"
  ☆24Updated last month
• davanstrien / haiku-dpo
  Using open source LLMs to build synthetic datasets for direct preference optimization
  ☆59Updated last year
• Pleias / Various-Finetuning
  Set of scripts to finetune LLMs
  ☆37Updated last year
• nahidalam / maya
  Maya: An Instruction Finetuned Multilingual Multimodal Model using Aya
  ☆107Updated last month
• AnswerDotAI / ModernBERT-Instruct-mini-cookbook
  ☆40Updated 2 months ago
• Knowledgator / GLiClass
  Generalist and Lightweight Model for Text Classification
  ☆115Updated this week
• ltgoslo / gpt-bert
  Official implementation of "GPT or BERT: why not both?"
  ☆52Updated last month
• Zyphra / Zyda_processing
  ☆33Updated 9 months ago
• ZurichNLP / mbr
  Minimum Bayes Risk Decoding for Hugging Face Transformers
  ☆57Updated 10 months ago
• cognitivecomputations / spectrum
  ☆112Updated this week
• facebookresearch / mexma
  MEXMA: Token-level objectives improve sentence representations
  ☆40Updated 3 months ago
• daniel-furman / sft-demos
  Lightweight demos for finetuning LLMs. Powered by 🤗 transformers and open-source datasets.
  ☆73Updated 5 months ago
• ContextualAI / CLAIR_and_APO
  Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  ☆55Updated 7 months ago
• louisbrulenaudet / ragoon
  High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡
  ☆67Updated 5 months ago
• huggingface / huggingface-inference-toolkit
  Hugging Face Inference Toolkit used to serve transformers, sentence-transformers, and diffusers models.
  ☆67Updated this week
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆50Updated last year
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆34Updated this week
• luyug / magix
  Supercharge huggingface transformers with model parallelism.
  ☆76Updated 6 months ago
• IBM / ensemble-instruct
  codebase release for EMNLP2023 paper publication
  ☆19Updated last year
• bminixhofer / zett
  Code for Zero-Shot Tokenizer Transfer
  ☆127Updated 3 months ago