Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• cadia-lvl / icelandic-NLP-resources
  Overview of Icelandic NLP resources at a glance
  ☆16Updated last year
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆255Updated 3 years ago
• tsproisl / SoMaJo
  A tokenizer and sentence splitter for German and English web and social media texts.
  ☆147Updated 11 months ago
• TurkuNLP / Turku-neural-parser-pipeline
  A neural parsing pipeline for segmentation, morphological tagging, dependency parsing and lemmatization with pre-trained models for more …
  ☆115Updated last year
• aboSamoor / pycld2
  ☆175Updated 7 months ago
• nreimers / truecaser
  Language independent truecaser in Python.
  ☆160Updated 4 years ago
• TakeLab / spacy-udpipe
  spaCy + UDPipe
  ☆163Updated 3 years ago
• Lilykos / pyphonetics
  A Python 3 phonetics library.
  ☆136Updated 5 years ago
• sorenlind / lemmy
  🤘Lemmy is a lemmatizer for Danish 🇩🇰 and Swedish 🇸🇪
  ☆79Updated 4 years ago
• tokestermw / spacy_hunspell
  Hunspell extension for spaCy 2.0.
  ☆94Updated last year
• allo-media / text2num
  Parse and convert numbers written in French, English, Spanish, Portuguese, German and Catalan into their digit representation.
  ☆111Updated 5 months ago
• fnl / syntok
  Text tokenization and sentence segmentation (segtok v2)
  ☆207Updated 3 years ago
• clips / dutchembeddings
  Repository for the word embeddings experiments described in "Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource", pre…
  ☆84Updated 4 years ago
• pyhunspell / pyhunspell
  (Official repo for pypi package) Python bindings for the Hunspell spellchecker engine
  ☆190Updated 4 years ago
• jonfd / nefnir
  A lemmatizer for Icelandic text
  ☆17Updated 7 years ago
• notAI-tech / deepsegment
  A sentence segmenter that actually works!
  ☆304Updated 5 years ago
• akshaynagpal / w2n
  Convert number words (eg. twenty one) to numeric digits (21)
  ☆180Updated 2 years ago
• pyconll / pyconll
  A minimal, pure Python library to interface with CoNLL-U format files.
  ☆152Updated this week
• cltl / OpenDutchWordnet
  This repo provides a python module to work with Open Dutch WordNet. It was created using python 3.4.
  ☆67Updated 4 years ago
• cfournie / segmentation.evaluation
  SegEval Segmentation Evaluation Package
  ☆56Updated 2 years ago
• clips / wordkit
  Featurize words into orthographic and phonological vectors.
  ☆41Updated 2 years ago
• daltonfury42 / truecase
  A python true casing utility that restores case information for texts
  ☆89Updated 3 years ago
• fnl / segtok
  Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic fe…
  ☆170Updated 3 years ago
• Abhijit-2592 / spacy-langdetect
  A fully customisable language detection pipeline for spaCy
  ☆93Updated 6 years ago
• R1j1t / contextualSpellCheck
  ✔️Contextual word checker for better suggestions (not actively maintained)
  ☆418Updated 9 months ago
• paulrinckens / timexy
  A spaCy custom component that extracts and normalizes temporal expressions
  ☆56Updated 2 years ago
• yannvgn / laserembeddings
  LASER multilingual sentence embeddings as a pip package
  ☆225Updated 2 years ago
• repodiac / german_compound_splitter
  Compound splitter for German language ("Komposita-Zerlegung") based on large dictionary combined with highly efficient multi-pattern stri…
  ☆33Updated 3 years ago
• explosion / spacy-lookups-data
  📂 Additional lookup tables and data resources for spaCy
  ☆112Updated 5 months ago