Alternatives and similar repositories for shiba

Users that are interested in shiba are comparing it to the libraries listed below

• shunk031 / allennlp-shiba-model

  View on GitHub
  AllenNLP integration for Shiba: Japanese CANINE model
  ☆12Jun 26, 2021Updated 4 years ago
• WorksApplications / SudachiTra

  View on GitHub
  Japanese tokenizer for Transformers
  ☆79Dec 15, 2023Updated 2 years ago
• chemicaltree / tetra

  View on GitHub
  ☆10Sep 14, 2022Updated 3 years ago
• megagonlabs / ginza-transformers

  View on GitHub
  Use custom tokenizers in spacy-transformers
  ☆16Aug 9, 2022Updated 3 years ago
• verypluming / JaNLI

  View on GitHub
  ☆17May 31, 2023Updated 2 years ago
• ku-nlp / AnnotatedFKCCorpus

  View on GitHub
  Annotated Fuman Kaitori Center Corpus
  ☆18Dec 18, 2023Updated 2 years ago
• yagays / ja-timex

  View on GitHub
  自然言語で書かれた時間情報表現を抽出/規格化するルールベースの解析器
  ☆140Feb 27, 2025Updated 11 months ago
• junya-takayama / DIRECT

  View on GitHub
  DIRECT: Direct and Indirect REsponses in Conversational Text Corpus
  ☆17Jul 1, 2021Updated 4 years ago
• azooKey / AJIMEE-Bench

  View on GitHub
  AJIMEE-Bench (Advanced Japanese IME Evaluation Benchmark)
  ☆18Jan 13, 2025Updated last year
• nttcslab / japanese-dialog-transformers

  View on GitHub
  Code for evaluating Japanese pretrained models provided by NTT Ltd.
  ☆245Jun 21, 2023Updated 2 years ago
• WorksApplications / chikkarpy

  View on GitHub
  Japanese synonym library
  ☆55Feb 7, 2022Updated 4 years ago
• himkt / awesome-bert-japanese

  View on GitHub
  📝 A list of pre-trained BERT models for Japanese with word/subword tokenization + vocabulary construction algorithm information
  ☆131Mar 15, 2023Updated 2 years ago
• megagonlabs / ebe-dataset

  View on GitHub
  Evidence-based Explanation Dataset (AACL-IJCNLP 2020)
  ☆18Dec 17, 2020Updated 5 years ago
• megagonlabs / jrte-corpus

  View on GitHub
  Japanese Realistic Textual Entailment Corpus (NLP 2020, LREC 2020)
  ☆77Jun 23, 2023Updated 2 years ago
• megagonlabs / bunkai

  View on GitHub
  Sentence boundary disambiguation tool for Japanese texts (日本語文境界判定器)
  ☆199Mar 26, 2024Updated last year
• HojiChar / HojiChar

  View on GitHub
  The robust text processing pipeline framework enabling customizable, efficient, and metric-logged text preprocessing.
  ☆125Nov 13, 2025Updated 3 months ago
• ou-medinfo / medbertjp

  View on GitHub
  Trials of pre-trained BERT models for the medical domain in Japanese.
  ☆12Nov 21, 2020Updated 5 years ago
• moisutsu / classopt

  View on GitHub
  Arguments parser with class for Python, inspired by StructOpt
  ☆62Sep 17, 2023Updated 2 years ago
• osekilab / JCoLA

  View on GitHub
  ☆19May 23, 2024Updated last year
• cl-tohoku / keigo_transfer_task

  View on GitHub
  敬語変換タスクにおける評価用データセット
  ☆21Nov 24, 2022Updated 3 years ago
• himkt / konoha

  View on GitHub
  🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.
  ☆261Apr 29, 2025Updated 9 months ago
• retarfi / language-pretraining

  View on GitHub
  Pre-training Language Models for Japanese
  ☆50Jul 2, 2023Updated 2 years ago
• ku-nlp / kwja

  View on GitHub
  An integrated Japanese analyzer based on foundation models
  ☆138Feb 2, 2026Updated 2 weeks ago
• ndl-lab / ndlngramdata

  View on GitHub
  デジタル化資料から作成したOCRテキストデータのngram頻度統計情報のデータセット
  ☆14Jan 10, 2023Updated 3 years ago
• izuna385 / Wikia-and-Wikipedia-EL-Dataset-Creator

  View on GitHub
  You can create datasets from Wikia/Wikipedia that can be used for entity recognition and Entity Linking. Dumps for ja-wiki and VTuber-wik…
  ☆17May 2, 2021Updated 4 years ago
• verypluming / JSICK

  View on GitHub
  Repository for JSICK
  ☆45May 31, 2023Updated 2 years ago
• wtsnjp / MioGatto

  View on GitHub
  An annotation tool for grounding of formulae
  ☆24May 28, 2024Updated last year
• yagays / nayose-wikipedia-ja

  View on GitHub
  Wikipediaから作成した日本語名寄せデータセット
  ☆35Mar 10, 2020Updated 5 years ago
• daac-tools / python-vaporetto

  View on GitHub
  🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.
  ☆21Jun 1, 2025Updated 8 months ago
• octanove / grammartagger

  View on GitHub
  GrammarTagger — A Neural Multilingual Grammar Profiler for Language Learning
  ☆31Apr 8, 2021Updated 4 years ago
• hppRC / defsent

  View on GitHub
  DefSent: Sentence Embeddings using Definition Sentences
  ☆22Aug 5, 2021Updated 4 years ago
• shigashiyama / nlp_survey

  View on GitHub
  ☆15Mar 31, 2020Updated 5 years ago
• SkelterLabsInc / JaQuAD

  View on GitHub
  JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension (2022, Skelter Labs)
  ☆108Mar 2, 2022Updated 3 years ago
• megagonlabs / UD_Japanese-GSD

  View on GitHub
  Japanese data from the Google UDT 2.0.
  ☆28Mar 24, 2023Updated 2 years ago
• sbintuitions / flexeval

  View on GitHub
  Flexible evaluation tool for language models
  ☆57Updated this week
• wwwcojp / ja_sentence_segmenter

  View on GitHub
  japanese sentence segmentation library for python
  ☆73Apr 3, 2023Updated 2 years ago
• ku-nlp / KWDLC

  View on GitHub
  Kyoto University Web Document Leads Corpus
  ☆83Dec 18, 2023Updated 2 years ago
• megagonlabs / t5-japanese

  View on GitHub
  Codes to pre-train Japanese T5 models
  ☆40Sep 7, 2021Updated 4 years ago
• laboroai / Laboro-ParaCorpus

  View on GitHub
  Scripts for creating a Japanese-English parallel corpus and training NMT models
  ☆18Nov 9, 2021Updated 4 years ago