Alternatives and similar repositories for tokenizers-wasm:

Users that are interested in tokenizers-wasm are comparing it to the libraries listed below

• FL33TW00D / embd
  GPU accelerated client-side embeddings for vector search, RAG etc.
  ☆66Updated last year
• Narsil / bloomserver
  ☆37Updated 2 years ago
• huggingface / ember
  ANE accelerated embedding models!
  ☆17Updated 3 months ago
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆25Updated last year
• iamlemec / bert.cpp
  GGML implementation of BERT model with Python bindings and quantization.
  ☆54Updated last year
• Narsil / hf-chat
  ☆25Updated 3 months ago
• LLukas22 / llm-rs-python
  Unofficial python bindings for the rust llm library. 🐍❤️🦀
  ☆75Updated last year
• Dan-wanna-M / kbnf
  A high-performance constrained decoding engine based on context free grammar in Rust
  ☆47Updated 2 months ago
• CarperAI / squeakily
  A library for squeakily cleaning and filtering language datasets.
  ☆46Updated last year
• castorini / hf-spacerini
  Plug-and-play Search Interfaces with Pyserini and Hugging Face
  ☆31Updated last year
• argilla-io / distilabel-spin-dibt
  Repository containing the SPIN experiments on the DIBT 10k ranked prompts
  ☆24Updated last year
• Narsil / fast_gpt2
  ☆153Updated 2 years ago
• RAIVNLab / AdANNS
  Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"
  ☆63Updated last year
• oxidized-transformers / oxidized-transformers
  Modular Rust transformer/LLM library using Candle
  ☆36Updated 10 months ago
• catid / lllm
  Latent Large Language Models
  ☆17Updated 6 months ago
• ArEnSc / Production-RWKV
  This project aims to make RWKV Accessible to everyone using a Hugging Face like interface, while keeping it close to the R and D RWKV bra…
  ☆64Updated last year
• mixedbread-ai / batched
  The Batched API provides a flexible and efficient way to process multiple requests in a batch, with a primary focus on dynamic batching o…
  ☆124Updated 2 months ago
• DeployQL / LintDB
  Vector Database with support for late interaction and token level embeddings.
  ☆53Updated 5 months ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 9 months ago
• hora-search / hora-wasm
  webassembly binding for Hora Approximate Nearest Neighbor Search Library
  ☆55Updated 3 years ago
• NolanoOrg / llama-int4-quant
  ☆26Updated 2 years ago
• UmerHA / triton_util
  Make triton easier
  ☆47Updated 9 months ago
• ashvardanian / usearch-binary
  Binary vector search example using Unum's USearch engine and pre-computed Wikipedia embeddings from Co:here and MixedBread
  ☆18Updated 11 months ago
• gsuuon / ad-llama
  Structured inference with Llama 2 in your browser
  ☆52Updated 4 months ago
• ggerganov / bert.cpp
  GGML implementation of BERT model with Python bindings and quantization.
  ☆24Updated last year
• ConiferLabsWA / flan-ul2-alpaca
  ☆32Updated last year
• huggingface / hffs
  **ARCHIVED** Filesystem interface to 🤗 Hub
  ☆58Updated last year