Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆255Updated 3 years ago
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• sorenlind / lemmy
  🤘Lemmy is a lemmatizer for Danish 🇩🇰 and Swedish 🇸🇪
  ☆79Updated 4 years ago
• tsproisl / SoMaJo
  A tokenizer and sentence splitter for German and English web and social media texts.
  ☆151Updated last year
• Lilykos / pyphonetics
  A Python 3 phonetics library.
  ☆138Updated 5 years ago
• nreimers / truecaser
  Language independent truecaser in Python.
  ☆160Updated 4 years ago
• adbar / simplemma
  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
  ☆184Updated 8 months ago
• fnl / syntok
  Text tokenization and sentence segmentation (segtok v2)
  ☆208Updated 3 years ago
• TakeLab / spacy-udpipe
  spaCy + UDPipe
  ☆166Updated 3 years ago
• TurkuNLP / Turku-neural-parser-pipeline
  A neural parsing pipeline for segmentation, morphological tagging, dependency parsing and lemmatization with pre-trained models for more …
  ☆115Updated last year
• cadia-lvl / icelandic-NLP-resources
  Overview of Icelandic NLP resources at a glance
  ☆18Updated last year
• aboSamoor / pycld2
  ☆178Updated 10 months ago
• allo-media / text2num
  Parse and convert numbers written in French, English, Spanish, Portuguese, German and Catalan into their digit representation.
  ☆112Updated 3 weeks ago
• fnl / segtok
  Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic fe…
  ☆171Updated 4 years ago
• globalwordnet / OMW
  The Open Multilingual Wordnet
  ☆66Updated last year
• ltgoslo / norne
  Norwegian Named Entities annotations on top of NDT (Norwegian Dependency Treebank)
  ☆71Updated last year
• daltonfury42 / truecase
  A python true casing utility that restores case information for texts
  ☆88Updated 3 years ago
• jacksonllee / pylangacq
  Language Acquisition Research Tools
  ☆43Updated 2 months ago
• explosion / spacy-lookups-data
  📂 Additional lookup tables and data resources for spaCy
  ☆113Updated 8 months ago
• yannvgn / laserembeddings
  LASER multilingual sentence embeddings as a pip package
  ☆224Updated 2 years ago
• aalto-speech / morfessor
  Morfessor is a tool for unsupervised and semi-supervised morphological segmentation
  ☆200Updated 5 years ago
• tokestermw / spacy_hunspell
  Hunspell extension for spaCy 2.0.
  ☆94Updated last year
• comtravo / ctparse
  Parse natural language time expressions in python
  ☆131Updated 3 years ago
• R1j1t / contextualSpellCheck
  ✔️Contextual word checker for better suggestions (not actively maintained)
  ☆418Updated last year
• argilla-io / spacy-wordnet
  spacy-wordnet creates annotations that easily allow the use of wordnet and wordnet domains by using the nltk wordnet interface
  ☆261Updated 5 months ago
• msg-systems / coreferee
  Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further lang…
  ☆197Updated 3 years ago
• pyconll / pyconll
  A minimal, pure Python library to interface with CoNLL-U format files.
  ☆153Updated 2 months ago
• dtuggener / CharSplit
  Compound splitter for German
  ☆111Updated 5 years ago
• proycon / python-frog
  Python bindings to the dutch NLP tool Frog (pos tagger, lemmatiser, NER tagger, morphological analysis, shallow parser, dependency parser…
  ☆49Updated this week
• pyhunspell / pyhunspell
  (Official repo for pypi package) Python bindings for the Hunspell spellchecker engine
  ☆189Updated 5 years ago