Alternatives and similar repositories for toxic-commons:

Users that are interested in toxic-commons are comparing it to the libraries listed below

• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆21Updated last year
• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆47Updated 3 weeks ago
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆22Updated 2 months ago
• gautierdag / tokenizer-bench
  Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"
  ☆15Updated last year
• salesforce / simplification
  ☆21Updated 3 weeks ago
• alvarobartt / vertex-ai-huggingface-inference-toolkit
  🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)
  ☆17Updated 11 months ago
• opinionscience / BERTransfer
  A BERT-based application for reusable text classification at scale
  ☆37Updated last year
• tylerachang / goldfish
  Goldfish: Monolingual language models for 350 languages.
  ☆15Updated 5 months ago
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆46Updated last year
• MilaNLProc / language-invariant-properties
  ☆22Updated 2 years ago
• pchizhov / picky_bpe
  BPE modification that implements removing of the intermediate tokens during tokenizer training.
  ☆25Updated 2 months ago
• Pleias / marginalia
  ☆67Updated 11 months ago
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• UKPLab / eacl2024-lagonn
  Source code and data for Like a Good Nearest Neighbor
  ☆28Updated last month
• nbroad1881 / strideformer
  Using short models to classify long texts
  ☆21Updated last year
• jackbandy / bookcorpus-datasheet
  Documentation effort for the BookCorpus dataset
  ☆33Updated 3 years ago
• malteos / llm-datasets
  A collection of datasets for language model pretraining including scripts for downloading, preprocesssing, and sampling.
  ☆55Updated 6 months ago
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆46Updated this week
• ielab / Starbucks
  Starbucks: Improved Training for 2D Matryoshka Embeddings
  ☆17Updated 2 weeks ago
• stefan-it / modern-bert-ner
  My NER Experiments with ModernBERT
  ☆17Updated last month
• Knowledgator / LiqFit
  Efficient few-shot learning with cross-encoders.
  ☆48Updated last year
• explosion / curated-tokenizers
  Lightweight piece tokenization library
  ☆12Updated 10 months ago
• lucy3 / whos_filtered
  ☆14Updated 4 months ago
• wjbmattingly / spacy-chunks
  An easy way to chunk spaCy docs.
  ☆19Updated 6 months ago
• guilhermemr04 / scaling-zero-shot-retrieval
  No Parameter Left Behind: How Distillation and Model Size Affect Zero-Shot Retrieval
  ☆28Updated 2 years ago
• argilla-io / distilabel-spin-dibt
  Repository containing the SPIN experiments on the DIBT 10k ranked prompts
  ☆24Updated 11 months ago
• malteos / clp-transfer
  Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
  ☆29Updated 2 years ago
• davanstrien / haiku-dpo
  Using open source LLMs to build synthetic datasets for direct preference optimization
  ☆57Updated 11 months ago
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆32Updated last year
• UniversalNER / UniversalNER
  ☆26Updated 6 months ago