mithril-security / tokenizers-wasmLinks
wasm bindings for huggingface tokenizers library
☆34Updated 3 years ago
Alternatives and similar repositories for tokenizers-wasm
Users that are interested in tokenizers-wasm are comparing it to the libraries listed below
Sorting:
- GPU accelerated client-side embeddings for vector search, RAG etc.☆65Updated last year
- Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and…☆34Updated 6 months ago
- ☆39Updated 3 years ago
- A complete(grpc service and lib) Rust inference with multilingual embedding support. This version leverages the power of Rust for both GR…☆39Updated last year
- A high-performance constrained decoding engine based on context free grammar in Rust☆54Updated 4 months ago
- High-performance MinHash implementation in Rust with Python bindings for efficient similarity estimation and deduplication of large datas…☆204Updated 2 months ago
- Simple high-throughput inference library☆142Updated 4 months ago
- GGML implementation of BERT model with Python bindings and quantization.☆55Updated last year
- Unofficial python bindings for the rust llm library. 🐍❤️🦀☆76Updated 2 years ago
- ☆157Updated 2 years ago
- Add local LLMs to your Web or Electron apps! Powered by Rust + WebGPU☆104Updated 2 years ago
- Web browser version of StarCoder.cpp☆45Updated 2 years ago
- Latent Large Language Models☆19Updated last year
- webassembly binding for Hora Approximate Nearest Neighbor Search Library☆58Updated 4 years ago
- Run ONNX and TensorFlow inference in the browser.☆75Updated 2 years ago
- Repository containing the SPIN experiments on the DIBT 10k ranked prompts☆24Updated last year
- ☆12Updated 8 months ago
- Optimizing bit-level Jaccard Index and Population Counts for large-scale quantized Vector Search via Harley-Seal CSA and Lookup Tables☆21Updated 4 months ago
- Inference Llama 2 in one file of zero-dependency, zero-unsafe Rust☆39Updated 2 years ago
- Experiments on speculative sampling with Llama models☆126Updated 2 years ago
- A sketch of a Transformer in Rust for a blog post☆34Updated 3 years ago
- Lightning Fast: Faiss CPU + Onnx Quantized Multilingual Embedding Model☆23Updated last year
- ☆58Updated 2 years ago
- Rust port of annoy (https://github.com/spotify/annoy)☆44Updated last month
- ☆26Updated 9 months ago
- Locality Sensitive Hashing☆73Updated 2 years ago
- ReLM is a Regular Expression engine for Language Models☆106Updated 2 years ago
- Code repository for the paper - "AdANNS: A Framework for Adaptive Semantic Search"☆65Updated last year
- Demonstration that finetuning RoPE model on larger sequences than the pre-trained model adapts the model context limit☆62Updated 2 years ago
- ☆133Updated last year