NovelAI / novelai-tokenizerView external linksLinks
Sentencepiece based BPE tokenizer for English and Japanese language text.
☆28Apr 4, 2024Updated last year
Alternatives and similar repositories for novelai-tokenizer
Users that are interested in novelai-tokenizer are comparing it to the libraries listed below
Sorting:
- Japanese instruction data (日本語指示データ)☆24Jul 13, 2023Updated 2 years ago
- Unofficial Knowledge Base for Novel AI☆15Nov 8, 2025Updated 3 months ago
- GPT2 Byte Pair Encoding implementation in Golang☆24Jul 9, 2025Updated 7 months ago
- Crawler and cleaner of data for novelai embedding's training☆21May 22, 2025Updated 8 months ago
- discord bot using AI to generate images based on discord messages☆11Oct 10, 2023Updated 2 years ago
- ☆24Dec 15, 2023Updated 2 years ago
- Karras et al. (2022) diffusion models for PyTorch☆17Oct 5, 2023Updated 2 years ago
- Trials of pre-trained BERT models for the medical domain in Japanese.☆12Nov 21, 2020Updated 5 years ago
- An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.☆13Jun 7, 2023Updated 2 years ago
- ☆17Sep 29, 2024Updated last year
- ☆16Nov 19, 2023Updated 2 years ago
- Training and evaluation scripts for JGLUE, a Japanese language understanding benchmark☆18Feb 2, 2026Updated last week
- A notebook running TensorRT's StableDiffusion demo on Google Colaboratory☆18Feb 1, 2023Updated 3 years ago
- An extension of cafe_aesthetic for AUTOMATIC1111's Stable Diffusion Web UI☆38Sep 14, 2023Updated 2 years ago
- A beamer template mainly for Japanese.☆14Apr 21, 2024Updated last year
- COrpus based Morphological Analyzer with INtegrated User dictionary☆21Mar 30, 2025Updated 10 months ago
- [ICML2025] Official Repo for Paper "Optimizing Temperature for Language Models with Multi-Sample Inference"☆21Feb 16, 2025Updated 11 months ago
- Annotated Fuman Kaitori Center Corpus☆18Dec 18, 2023Updated 2 years ago
- vits☆21Mar 29, 2023Updated 2 years ago
- The repository contains scripts and merge scripts that have been modified to adapt an Alpaca-Lora adapter for LoRA tuning when assuming t…☆18May 24, 2023Updated 2 years ago
- Repository for the code assignment of the Deep Learning 1 course, Fall 2022 edition☆20Dec 9, 2022Updated 3 years ago
- LLM構築用の日本語チャットデータセット☆87Jan 23, 2024Updated 2 years ago
- A re-implementation of Stable-Diffusion using better code pratices with faster and lower-memory usage.☆45Feb 8, 2023Updated 3 years ago
- ☆39Sep 24, 2025Updated 4 months ago
- 敬語変換タスクにおける評価用データセット☆21Nov 24, 2022Updated 3 years ago
- 44100Hz日本語音源に対応させた unofficial vits2-TTS implementation in pytorchです。☆24Sep 1, 2023Updated 2 years ago
- Platform and API Agnostic library for powering chatbots☆24Feb 27, 2023Updated 2 years ago
- Implementation of MambaFormer in Pytorch ++ Zeta from the paper: "Can Mamba Learn How to Learn? A Comparative Study on In-Context Learnin…☆21Updated this week
- Flexible evaluation tool for language models☆57Jan 28, 2026Updated 2 weeks ago
- A notebook-based web UI for DeepFloyd IF☆25May 31, 2024Updated last year
- Benchmark for Japanese document embedding & vector search☆29Mar 12, 2024Updated last year
- Checkpointable dataset utilities for foundation model training☆32Jan 29, 2024Updated 2 years ago
- ☆28Nov 23, 2023Updated 2 years ago
- RealPersonaChat: A Realistic Persona Chat Corpus with Interlocutors' Own Personalities☆63Mar 13, 2024Updated last year
- Japanese data from the Google UDT 2.0.☆28Mar 24, 2023Updated 2 years ago
- Aesthetic classifier training & inference tools☆31Nov 6, 2022Updated 3 years ago
- CyberAgent AI Lab研修： "モデルコードの高速化・最適化チュートリアル"☆35Mar 13, 2025Updated 11 months ago
- ☆12Sep 13, 2025Updated 5 months ago
- Utility scripts for preprocessing Wikipedia texts for NLP☆78Apr 9, 2024Updated last year