Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• tsproisl / SoMaJo
  A tokenizer and sentence splitter for German and English web and social media texts.
  ☆150Updated last year
• Lilykos / pyphonetics
  A Python 3 phonetics library.
  ☆137Updated 5 years ago
• nreimers / truecaser
  Language independent truecaser in Python.
  ☆160Updated 4 years ago
• allo-media / text2num
  Parse and convert numbers written in French, English, Spanish, Portuguese, German and Catalan into their digit representation.
  ☆112Updated 2 weeks ago
• TakeLab / spacy-udpipe
  spaCy + UDPipe
  ☆166Updated 3 years ago
• fnl / syntok
  Text tokenization and sentence segmentation (segtok v2)
  ☆208Updated 3 years ago
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆256Updated 3 years ago
• TurkuNLP / Turku-neural-parser-pipeline
  A neural parsing pipeline for segmentation, morphological tagging, dependency parsing and lemmatization with pre-trained models for more …
  ☆115Updated last year
• sorenlind / lemmy
  🤘Lemmy is a lemmatizer for Danish 🇩🇰 and Swedish 🇸🇪
  ☆79Updated 4 years ago
• repodiac / german_compound_splitter
  Compound splitter for German language ("Komposita-Zerlegung") based on large dictionary combined with highly efficient multi-pattern stri…
  ☆34Updated 3 years ago
• chrislit / abydos
  Abydos NLP/IR library for Python
  ☆194Updated 3 years ago
• cadia-lvl / icelandic-NLP-resources
  Overview of Icelandic NLP resources at a glance
  ☆16Updated last year
• jonfd / nefnir
  A lemmatizer for Icelandic text
  ☆17Updated 7 years ago
• Kozea / Pyphen
  Hy-phen-ation made easy
  ☆218Updated 3 weeks ago
• pyhunspell / pyhunspell
  (Official repo for pypi package) Python bindings for the Hunspell spellchecker engine
  ☆189Updated 4 years ago
• explosion / spacy-lookups-data
  📂 Additional lookup tables and data resources for spaCy
  ☆113Updated 7 months ago
• tokestermw / spacy_hunspell
  Hunspell extension for spaCy 2.0.
  ☆94Updated last year
• R1j1t / contextualSpellCheck
  ✔️Contextual word checker for better suggestions (not actively maintained)
  ☆417Updated 11 months ago
• flammie / omorfi
  Open morphology for Finnish
  ☆95Updated last week
• argilla-io / spacy-wordnet
  spacy-wordnet creates annotations that easily allow the use of wordnet and wordnet domains by using the nltk wordnet interface
  ☆261Updated 5 months ago
• aalto-speech / morfessor
  Morfessor is a tool for unsupervised and semi-supervised morphological segmentation
  ☆200Updated 5 years ago
• aboSamoor / pycld2
  ☆177Updated 10 months ago
• pyenchant / pyenchant
  spellchecking library for python
  ☆617Updated 4 months ago
• davidberenstein1957 / classy-classification
  This repository contains an easy and intuitive approach to few-shot classification using sentence-transformers or spaCy models, or zero-s…
  ☆220Updated last year
• adbar / simplemma
  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
  ☆182Updated 7 months ago
• msg-systems / coreferee
  Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further lang…
  ☆197Updated 3 years ago
• mhulden / pyfoma
  Python Finite-State Toolkit
  ☆60Updated last month
• SekouD / mlconjug
  A Python library to conjugate verbs in French, English, Spanish, Italian, Portuguese and Romanian (more soon) using Machine Learning tech…
  ☆74Updated last year
• fnl / segtok
  Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic fe…
  ☆171Updated 4 years ago