Sentencepiece based BPE tokenizer for English and Japanese language text.
☆28Apr 4, 2024Updated last year
Alternatives and similar repositories for novelai-tokenizer
Users that are interested in novelai-tokenizer are comparing it to the libraries listed below
Sorting:
- Japanese instruction data (日本語指示データ)☆24Jul 13, 2023Updated 2 years ago
- ☆33Jul 31, 2024Updated last year
- GPT2 Byte Pair Encoding implementation in Golang☆24Jul 9, 2025Updated 7 months ago
- Unofficial Knowledge Base for Novel AI☆16Nov 8, 2025Updated 3 months ago
- discord bot using AI to generate images based on discord messages☆11Oct 10, 2023Updated 2 years ago
- ☆24Dec 15, 2023Updated 2 years ago
- Meta data extraction scripts for images generated with NovelAI's image generation☆35Feb 4, 2026Updated last month
- Trials of pre-trained BERT models for the medical domain in Japanese.☆12Nov 21, 2020Updated 5 years ago
- Karras et al. (2022) diffusion models for PyTorch☆17Oct 5, 2023Updated 2 years ago
- An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.☆13Jun 7, 2023Updated 2 years ago
- ☆16Mar 4, 2024Updated 2 years ago
- A magic notepad. δ☆14May 21, 2023Updated 2 years ago
- NovelAI 5ch wikiの拡張jsのリポジトリです☆12Oct 16, 2022Updated 3 years ago
- ☆42Jun 22, 2023Updated 2 years ago
- ☆16Nov 19, 2023Updated 2 years ago
- A notebook running TensorRT's StableDiffusion demo on Google Colaboratory☆18Feb 1, 2023Updated 3 years ago
- ☆18Sep 29, 2024Updated last year
- Training and evaluation scripts for JGLUE, a Japanese language understanding benchmark☆18Updated this week
- COrpus based Morphological Analyzer with INtegrated User dictionary☆21Mar 30, 2025Updated 11 months ago
- [ICML2025] Official Repo for Paper "Optimizing Temperature for Language Models with Multi-Sample Inference"☆21Feb 16, 2025Updated last year
- The repository contains scripts and merge scripts that have been modified to adapt an Alpaca-Lora adapter for LoRA tuning when assuming t…☆18May 24, 2023Updated 2 years ago
- Annotated Fuman Kaitori Center Corpus☆18Dec 18, 2023Updated 2 years ago
- Repository for the code assignment of the Deep Learning 1 course, Fall 2022 edition☆20Dec 9, 2022Updated 3 years ago
- vits☆21Mar 29, 2023Updated 2 years ago
- LLM構築用の日本語チャットデータセット☆88Jan 23, 2024Updated 2 years ago
- A re-implementation of Stable-Diffusion using better code pratices with faster and lower-memory usage.☆45Feb 8, 2023Updated 3 years ago
- 敬語変換タスクにおける評価用データセット☆21Nov 24, 2022Updated 3 years ago
- 44100Hz日本語音源に対応させた unofficial vits2-TTS implementation in pytorchです。☆24Sep 1, 2023Updated 2 years ago
- ☆40Feb 25, 2026Updated last week
- Platform and API Agnostic library for powering chatbots☆24Feb 27, 2023Updated 3 years ago
- python版日本語意味役割付与システム（ASA）☆22Nov 11, 2022Updated 3 years ago
- Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learnin…☆21Feb 9, 2026Updated 3 weeks ago
- Benchmark for Japanese document embedding & vector search☆29Mar 12, 2024Updated last year
- A notebook-based web UI for DeepFloyd IF☆25May 31, 2024Updated last year
- Flexible evaluation tool for language models☆58Feb 26, 2026Updated last week
- ☆28Nov 23, 2023Updated 2 years ago
- RealPersonaChat: A Realistic Persona Chat Corpus with Interlocutors' Own Personalities☆63Mar 13, 2024Updated last year
- Webブラウザから手軽にローカルLLMとおしゃべりできるソフトウェアです。☆27Dec 28, 2023Updated 2 years ago
- Japanese data from the Google UDT 2.0.☆28Mar 24, 2023Updated 2 years ago