Alternatives and similar repositories for curated-tokenizers:

Users that are interested in curated-tokenizers are comparing it to the libraries listed below

• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆34Updated this week
• kpu / fasterText
  Library for fast text representation and classification.
  ☆28Updated last year
• bltlab / seqscore
  SeqScore: Scoring for named entity recognition and other sequence labeling tasks
  ☆22Updated 2 weeks ago
• thakur-nandan / income
  INCOME: An Easy Repository for Training and Evaluation of Index Compression Methods in Dense Retrieval. Includes BPR and JPQ.
  ☆22Updated last year
• UKPLab / eacl2024-lagonn
  Source code and data for Like a Good Nearest Neighbor
  ☆28Updated last week
• sophiaalthammer / parm
  This repository contains the code for the paper 'PARM: Paragraph Aggregation Retrieval Model for Dense Document-to-Document Retrieval' pu…
  ☆40Updated 3 years ago
• argilla-io / adept-augmentations
  A Python library aimed at dissecting and augmenting NER training data.
  ☆57Updated last year
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆22Updated last month
• ielab / Starbucks
  Starbucks: Improved Training for 2D Matryoshka Embeddings
  ☆17Updated 3 months ago
• explosion / spacy-experimental
  🧪 Cutting-edge experimental spaCy components and features
  ☆96Updated 8 months ago
• KennethEnevoldsen / spacy-wrap
  spaCy-wrap is a wrapper library for spaCy for including fine-tuned transformers from Huggingface in your spaCy pipeline allowing you to i…
  ☆46Updated 9 months ago
• explosion / wikid
  Generate a SQLite database from Wikipedia & Wikidata dumps.
  ☆30Updated 9 months ago
• paulrinckens / timexy
  A spaCy custom component that extracts and normalizes temporal expressions
  ☆52Updated last year
• BramVanroy / astred
  An easy-to-use library to linguistically compare one sentence and its words to another, in the same language or a different one. For inst…
  ☆22Updated 3 years ago
• valentinhofmann / flota
  ☆17Updated last year
• salesforce / simplification
  ☆21Updated last year
• google-research-datasets / swim-ir
  SWIM-IR is a Synthetic Wikipedia-based Multilingual Information Retrieval training set with 28 million query-passage pairs spanning 33 la…
  ☆45Updated last year
• machamp-nlp / machamp
  Repository with code for MaChAmp: https://aclanthology.org/2021.eacl-demos.22/
  ☆83Updated last week
• helboukkouri / character-bert-pretraining
  Code for pre-training CharacterBERT models (as well as BERT models).
  ☆34Updated 3 years ago
• webis-de / lightning-ir
  One-stop shop for running and fine-tuning transformer-based language models for retrieval
  ☆43Updated this week
• tigerchen52 / GLADIS
  GLADIS: A General and Large Acronym Disambiguation Benchmark (EACL 23)
  ☆13Updated 6 months ago
• gautierdag / tokenizer-bench
  Code for the paper "Getting the most out of your tokenizer for pre-training and domain adaptation"
  ☆14Updated 11 months ago
• lucy3 / whos_filtered
  ☆14Updated 3 months ago
• dhfbk / KIND
  KIND: an Italian Multi-Domain Dataset for Named Entity Recognition
  ☆15Updated last year
• allenai / pybart
  Converter from UD-trees to BART representation
  ☆36Updated 10 months ago
• UniversalNER / UniversalNER
  ☆26Updated 5 months ago
• stephantul / piecelearn
  Learning BPE embeddings by first learning a segmentation model and then training word2vec
  ☆19Updated 2 years ago
• flipz357 / S3BERT
  Semantically Structured Sentence Embeddings
  ☆66Updated 3 months ago
• marian-nmt / sotastream
  A library for data streaming and augmentation
  ☆20Updated 10 months ago
• NathanGodey / headless-lm
  Training and evaluation code for the paper "Headless Language Models: Learning without Predicting with Contrastive Weight Tying" (https:/…
  ☆24Updated 9 months ago