Alternatives and similar repositories for fast_bpe_tokenizer:

Users that are interested in fast_bpe_tokenizer are comparing it to the libraries listed below

• Jellyfish042 / uncheatable_eval
  Evaluating LLMs with Dynamic Data
  ☆77Updated last month
• BlinkDL / modded-nanogpt-rwkv
  RWKV-7: Surpassing GPT
  ☆80Updated 3 months ago
• NormXU / Consistent-DynamicNTKRoPE
  An Experiment on Dynamic NTK Scaling RoPE
  ☆62Updated last year
• thu-ml / low-bit-optimizers
  Low-bit optimizers for PyTorch
  ☆125Updated last year
• liyucheng09 / llm-compressive
  Longitudinal Evaluation of LLMs via Data Compression
  ☆32Updated 9 months ago
• 00ffcc / chunkRWKV6
  continous batching and parallel acceleration for RWKV6
  ☆24Updated 8 months ago
• kyegomez / Blockwise-Parallel-Transformer
  32 times longer context window than vanilla Transformers and up to 4 times longer than memory efficient Transformers.
  ☆46Updated last year
• shaochenze / PatchTrain
  Code for paper "Patch-Level Training for Large Language Models"
  ☆81Updated 3 months ago
• thunlp / Ouroboros
  Ouroboros: Speculative Decoding with Large Model Enhanced Drafting (EMNLP 2024 main)
  ☆85Updated 5 months ago
• DAMO-NLP-SG / CLEX
  [ICLR 2024] CLEX: Continuous Length Extrapolation for Large Language Models
  ☆76Updated last year
• NolanoOrg / sparse_quant_llms
  SparseGPT + GPTQ Compression of LLMs like LLaMa, OPT, Pythia
  ☆41Updated last year
• kyegomez / Infini-attention
  Implementation of the paper: "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" from Google in pyTO…
  ☆53Updated last month
• tridao / flash-attention-wheels
  ☆46Updated last year
• Lightning-Universe / lightning-ColossalAI
  Large Scale Distributed Model Training strategy with Colossal AI and Lightning AI
  ☆57Updated last year
• BBuf / flash-rwkv
  ☆30Updated 9 months ago
• YuchuanTian / DiJiang
  [ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear at…
  ☆99Updated 8 months ago
• OpenNLPLab / LASP
  Linear Attention Sequence Parallelism (LASP)
  ☆79Updated 9 months ago
• dust-tt / llama-ssp
  Experiments on speculative sampling with Llama models
  ☆125Updated last year
• BBuf / RWKV-World-HF-Tokenizer
  ☆34Updated 7 months ago
• horseee / LLaMA-Pruning
  Structural Pruning for LLaMA
  ☆54Updated last year
• BlinkDL / nanoRWKV
  RWKV in nanoGPT style
  ☆187Updated 9 months ago
• Nicolas-BZRD / llm-recipes
  ☆28Updated 11 months ago
• gf712 / gpt2-cpp
  GPT2 implementation in C++ using Ort
  ☆26Updated 4 years ago
• zankner / Hydra
  ☆42Updated last year
• feifeibear / Odysseus-Transformer
  Odysseus: Playground of LLM Sequence Parallelism
  ☆66Updated 8 months ago
• RyokoAI / BigKnow2022
  BigKnow2022: Bringing Language Models Up to Speed
  ☆14Updated last year
• dwzhu-pku / PoSE
  Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)
  ☆204Updated 9 months ago
• chu-tianxiang / QuIP-for-all
  QuIP quantization
  ☆51Updated 11 months ago
• FasterDecoding / TEAL
  ☆115Updated 3 weeks ago