Alternatives and similar repositories for Tokenizer-Evaluation

Users that are interested in Tokenizer-Evaluation are comparing it to the libraries listed below

• MinishLab / tokenlearn
  Pre-train Static Word Embeddings
  ☆79Updated 3 weeks ago
• commoncrawl / web-languages
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ …
  ☆45Updated this week
• kpu / fasterText
  Library for fast text representation and classification.
  ☆30Updated last year
• imohitmayank / sentenceviz
  A Streamlit application to visualize sentence embeddings
  ☆19Updated 2 years ago
• oxidized-transformers / oxidized-transformers
  Modular Rust transformer/LLM library using Candle
  ☆36Updated last year
• bitextor / warc2text
  Extracts plain text, language identification and more metadata from WARC records
  ☆22Updated 3 months ago
• flairNLP / fabricator
  [EMNLP 2023 Demo] fabricator - annotating and generating datasets with large language models.
  ☆109Updated last year
• MeLeLBGU / SaGe
  Code for SaGe subword tokenizer (EACL 2023)
  ☆25Updated 6 months ago
• maxdotio / mighty-batch
  Highly concurrent and fast content processing for Mighty Inference Server
  ☆10Updated 2 years ago
• wikimedia / sentencex
  A sentence segmentation library with wide language support optimized for speed and utility.
  ☆65Updated this week
• browsermt / marian-dev
  Fast Neural Machine Translation in C++ - development repository
  ☆19Updated last year
• alea-institute / nupunkt
  Next-generation Punkt sentence boundary detection with zero dependencies
  ☆17Updated 2 months ago
• Halvani / alphabetic
  A Python module for retrieving script types of writing systems including alphabets, abjads, abugidas, syllabaries, logographs, featurals …
  ☆13Updated 11 months ago
• raphaelsty / LeNLP
  NLP with Rust for Python 🦀🐍
  ☆62Updated last month
• UBC-NLP / afrolid
  AfroLID, a powerful neural toolkit for African languages identification which covers 517 African languages.
  ☆31Updated 3 months ago
• ltgoslo / ltg-bert
  LTG-Bert
  ☆33Updated last year
• proycon / analiticcl
  an approximate string matching or fuzzy-matching system for spelling correction, normalisation or post-OCR correction
  ☆37Updated 3 months ago
• taylorai / onnx_embedding_models
  utilities for loading and running text embeddings with onnx
  ☆44Updated 10 months ago
• Knowledgator / FlashDeBERTa
  Trully flash implementation of DeBERTa disentangled attention mechanism.
  ☆58Updated last month
• neuml / staticvectors
  🔢 Work with static vector models
  ☆28Updated 2 months ago
• mhulden / pyfoma
  Python Finite-State Toolkit
  ☆56Updated this week
• alvarobartt / vertex-ai-huggingface-inference-toolkit
  🤗 HuggingFace Inference Toolkit for Google Cloud Vertex AI (similar to SageMaker's Inference Toolkit, but for Vertex AI and unofficial)
  ☆17Updated last year
• Pleias / Pleias-RAG-Library
  Python library to use Pleias-RAG models
  ☆57Updated last month
• aixplain / aiXplain
  aiXplain enables python programmers to add AI functions to their software.
  ☆44Updated this week
• morrisalp / taatiknet
  Character-level conversion between Hebrew text and Latin transliteration using deep learning - a demonstration of seq2seq training.
  ☆13Updated last year
• explosion / spacy-huggingface-pipelines
  💥 Use Hugging Face text and token classification pipelines directly in spaCy
  ☆63Updated last year
• nateraw / huggingface-datasets-converter
  Scripts to convert datasets from various sources to Hugging Face Datasets.
  ☆57Updated 2 years ago
• EleutherAI / tokengrams
  Efficiently computing & storing token n-grams from large corpora
  ☆24Updated 8 months ago
• mixedbread-ai / binary-embeddings
  Showcase how mxbai-embed-large-v1 can be used to produce binary embedding. Binary embeddings enabled 32x storage savings and 40x faster r…
  ☆18Updated last year
• cmakafui / batchwizard
  A CLI tool for managing OpenAI batch processing jobs with ease.
  ☆37Updated last month