Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• allo-media / text2num
  Parse and convert numbers written in French, English, Spanish, Portuguese, German and Catalan into their digit representation.
  ☆112Updated 3 weeks ago
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆255Updated 3 years ago
• nreimers / truecaser
  Language independent truecaser in Python.
  ☆160Updated 4 years ago
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• cadia-lvl / icelandic-NLP-resources
  Overview of Icelandic NLP resources at a glance
  ☆18Updated last year
• TakeLab / spacy-udpipe
  spaCy + UDPipe
  ☆166Updated 3 years ago
• TurkuNLP / Turku-neural-parser-pipeline
  A neural parsing pipeline for segmentation, morphological tagging, dependency parsing and lemmatization with pre-trained models for more …
  ☆115Updated last year
• aalto-speech / morfessor
  Morfessor is a tool for unsupervised and semi-supervised morphological segmentation
  ☆200Updated 5 years ago
• abuccts / wikt2pron
  A Python toolkit converting pronunciation in enwiktionary xml dump to cmudict format
  ☆33Updated 6 years ago
• pyconll / pyconll
  A minimal, pure Python library to interface with CoNLL-U format files.
  ☆153Updated 2 months ago
• akshaynagpal / w2n
  Convert number words (eg. twenty one) to numeric digits (21)
  ☆180Updated 2 years ago
• Lilykos / pyphonetics
  A Python 3 phonetics library.
  ☆138Updated 5 years ago
• fnl / syntok
  Text tokenization and sentence segmentation (segtok v2)
  ☆208Updated 3 years ago
• cfournie / segmentation.evaluation
  SegEval Segmentation Evaluation Package
  ☆57Updated 2 years ago
• dtuggener / CharSplit
  Compound splitter for German
  ☆111Updated 5 years ago
• fnl / segtok
  Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic fe…
  ☆171Updated 4 years ago
• tsproisl / SoMaJo
  A tokenizer and sentence splitter for German and English web and social media texts.
  ☆151Updated last year
• adbar / simplemma
  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
  ☆184Updated 8 months ago
• udapi / udapi-python
  Python framework for processing Universal Dependencies data
  ☆59Updated last week
• lingpy / lingpy
  LingPy: Python library for quantitative tasks in historical linguistics
  ☆139Updated 2 months ago
• jonfd / nefnir
  A lemmatizer for Icelandic text
  ☆17Updated 7 years ago
• wroberts / pygermanet
  GermaNet API for Python
  ☆54Updated 7 years ago
• prosegrinder / python-cmudict
  A versioned python wrapper package for cmudict (https://github.com/cmusphinx/cmudict).
  ☆67Updated this week
• notAI-tech / deepsegment
  A sentence segmenter that actually works!
  ☆304Updated 5 years ago
• mhulden / pyfoma
  Python Finite-State Toolkit
  ☆60Updated last month
• aboSamoor / pycld2
  ☆178Updated 10 months ago
• clips / wordkit
  Featurize words into orthographic and phonological vectors.
  ☆41Updated 2 years ago
• AU-DIS / LSTM_langid
  Source code for the Apple reproduction
  ☆33Updated 4 years ago
• R1j1t / contextualSpellCheck
  ✔️Contextual word checker for better suggestions (not actively maintained)
  ☆418Updated last year
• bedapudi6788 / deepcorrect
  Text and Punctuation correction with Deep Learning
  ☆128Updated 5 years ago