Alternatives and similar repositories for bytepiece-rs

Users that are interested in bytepiece-rs are comparing it to the libraries listed below

• SunDoge / bytepiece-rs
  更纯粹、更高压缩率的Tokenizer in Rust
  ☆12Updated 5 months ago
• mrsteyk / RWKV-LM-deepspeed
  ☆42Updated 2 years ago
• LaurentMazare / glim
  ☆19Updated 8 months ago
• mrsteyk / rwkvk-rs
  ☆32Updated 2 years ago
• serega / gaoya
  Locality Sensitive Hashing
  ☆71Updated last year
• haixuanTao / bert-onnx-rs-server
  A Demo server serving Bert through ONNX with GPU written in Rust with <3
  ☆40Updated 3 years ago
• FreedomIntelligence / FastLLM
  Fast LLM Training CodeBase With dynamic strategy choosing [Deepspeed+Megatron+FlashAttention+CudaFusionKernel+Compiler];
  ☆37Updated last year
• Narsil / bloomserver
  ☆39Updated 2 years ago
• EricLBuehler / float8
  8-bit floating point types for Rust
  ☆45Updated 2 months ago
• Dan-wanna-M / kbnf
  A high-performance constrained decoding engine based on context free grammar in Rust
  ☆53Updated 2 weeks ago
• ljt019 / transformers
  Transformers provides a simple, intuitive interface for Rust developers who want to work with Large Language Models locally, powered by t…
  ☆16Updated this week
• eugenehp / gpu-fft
  GPU based FFT written in Rust and CubeCL
  ☆22Updated 2 months ago
• chenwanqq / candle-llava
  implement llava using candle
  ☆15Updated 11 months ago
• beichao1314 / Open-Llama
  The complete training code of the open-source high-performance Llama model, including the full process from pre-training to RLHF.
  ☆65Updated 2 years ago
• onesuper / HuggingFace-Datasets-Text-Quality-Analysis
  Retrieves parquet files from Hugging Face, identifies and quantifies junky data, duplication, contamination, and biased content in datase…
  ☆53Updated last year
• Oneflow-Inc / one-glm
  A more efficient GLM implementation!
  ☆55Updated 2 years ago
• HITsz-TMG / KaLM-Embedding
  Code for KaLM-Embedding models
  ☆78Updated 2 months ago
• RWKV-Wiki / MultilingualShareGPT
  MultilingualShareGPT, the free multi-language corpus for LLM training
  ☆72Updated 2 years ago
• jiahe7ay / infini-mini-transformer
  This is a personal reimplementation of Google's Infini-transformer, utilizing a small 2b model. The project includes both model and train…
  ☆56Updated last year
• iamlemec / bert.cpp
  GGML implementation of BERT model with Python bindings and quantization.
  ☆55Updated last year
• gnp / minbpe-rs
  Port of Andrej Karpathy's minbpe to Rust
  ☆23Updated last year
• LC1332 / Luotuo-Silk-Road
  Silk Road will be the dataset zoo for Luotuo(骆驼). Luotuo is an open sourced Chinese-LLM project founded by 陈启源 @ 华中师范大学 & 李鲁鲁 @ 商汤科技 & 冷子…
  ☆39Updated last year
• hemingkx / SpecDec
  Codes for our paper "Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation" (EMNLP 2023 Findings)
  ☆41Updated last year
• Harry-Chen / InfMoE
  Inference framework for MoE layers based on TensorRT with Python binding
  ☆41Updated 4 years ago
• nkypy / candle-rwkv
  RWKV models and examples powered by candle.
  ☆18Updated 3 months ago
• Academic-Hammer / HammerLLM
  1.4B sLLM for Chinese and English - HammerLLM🔨
  ☆44Updated last year
• kyutai-labs / kaudio
  Rust crate for some audio utilities
  ☆23Updated 3 months ago
• THUDM / Multilingual-GLM
  The multilingual variant of GLM, a general language model trained with autoregressive blank infilling objective
  ☆62Updated 2 years ago
• vllm-project / vllm-nccl
  Manages vllm-nccl dependency
  ☆17Updated last year
• LaurentMazare / tch-ext
  Sample Python extension using Rust/PyO3/tch to interact with PyTorch
  ☆37Updated last year