Alternatives and similar repositories for Tokenizer

Users that are interested in Tokenizer are comparing it to the libraries listed below

• sorenlind / lemmy
  🤘Lemmy is a lemmatizer for Danish 🇩🇰 and Swedish 🇸🇪
  ☆77Updated 3 years ago
• tsproisl / SoMaJo
  A tokenizer and sentence splitter for German and English web and social media texts.
  ☆147Updated 8 months ago
• TakeLab / spacy-udpipe
  spaCy + UDPipe
  ☆163Updated 3 years ago
• TurkuNLP / Turku-neural-parser-pipeline
  A neural parsing pipeline for segmentation, morphological tagging, dependency parsing and lemmatization with pre-trained models for more …
  ☆113Updated last year
• nreimers / truecaser
  Language independent truecaser in Python.
  ☆159Updated 3 years ago
• MSeal / cython_hunspell
  Cython wrapper on Hunspell Dictionary
  ☆66Updated last year
• mediacloud / sentence-splitter
  Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
  ☆251Updated 2 years ago
• repodiac / german_compound_splitter
  Compound splitter for German language ("Komposita-Zerlegung") based on large dictionary combined with highly efficient multi-pattern stri…
  ☆29Updated 3 years ago
• Lilykos / pyphonetics
  A Python 3 phonetics library.
  ☆133Updated 5 years ago
• aboSamoor / pycld2
  ☆172Updated 4 months ago
• fnl / syntok
  Text tokenization and sentence segmentation (segtok v2)
  ☆205Updated 3 years ago
• explosion / spacy-lookups-data
  📂 Additional lookup tables and data resources for spaCy
  ☆108Updated 2 months ago
• cadia-lvl / icelandic-NLP-resources
  Overview of Icelandic NLP resources at a glance
  ☆16Updated last year
• allo-media / text2num
  Parse and convert numbers written in French, English, Spanish, Portuguese, German and Catalan into their digit representation.
  ☆107Updated 2 months ago
• omwn / omw-data
  This packages up data for the Open Multilingual Wordnet
  ☆50Updated 2 months ago
• hdaSprachtechnologie / odenet
  Open German WordNet
  ☆96Updated last year
• fnl / segtok
  Segtok v2 is here: https://github.com/fnl/syntok -- A rule-based sentence segmenter (splitter) and a word tokenizer using orthographic fe…
  ☆170Updated 3 years ago
• clips / dutchembeddings
  Repository for the word embeddings experiments described in "Evaluating Unsupervised Dutch Word Embeddings as a Linguistic Resource", pre…
  ☆83Updated 4 years ago
• proycon / python-frog
  Python bindings to the dutch NLP tool Frog (pos tagger, lemmatiser, NER tagger, morphological analysis, shallow parser, dependency parser…
  ☆49Updated 4 months ago
• globalwordnet / OMW
  The Open Multilingual Wordnet
  ☆63Updated last year
• dtuggener / CharSplit
  Compound splitter for German
  ☆108Updated 5 years ago
• ltgoslo / norne
  Norwegian Named Entities annotations on top of NDT (Norwegian Dependency Treebank)
  ☆69Updated 11 months ago
• adbar / simplemma
  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
  ☆168Updated 2 months ago
• miotto / treetagger-python
  A Python module for interfacing with the Treetagger by Helmut Schmid.
  ☆75Updated 2 months ago
• TimKam / compound-word-splitter
  A compound word splitter for Python
  ☆48Updated 3 years ago
• chrislit / abydos
  Abydos NLP/IR library for Python
  ☆188Updated 2 years ago
• tokestermw / spacy_hunspell
  Hunspell extension for spaCy 2.0.
  ☆94Updated last year
• wroberts / pygermanet
  GermaNet API for Python
  ☆53Updated 7 years ago
• cfournie / segmentation.evaluation
  SegEval Segmentation Evaluation Package
  ☆56Updated 2 years ago
• argilla-io / spacy-wordnet
  spacy-wordnet creates annotations that easily allow the use of wordnet and wordnet domains by using the nltk wordnet interface
  ☆260Updated 11 months ago