Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• go-skynet / go-ggml-transformers.cpp
  Binding to transformers in ggml
  ☆63Updated this week
• milosgajdos / embeviz
  A simple app for visualising text embeddings
  ☆21Updated 3 months ago
• shoenig / go-landlock
  Go filesystem isolation via Linux landlock sandbox
  ☆17Updated 4 months ago
• pjbgf / sha1cd
  SHA1 implementation with collision detection.
  ☆13Updated last week
• c0mm4nd / wasman
  Another wasm interpreter engine for gophers based on pure golang
  ☆11Updated 3 years ago
• gotzmann / booster
  Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
  ☆158Updated 11 months ago
• Kurt212 / syncgroup
  Golang goroutines synchronization primitive wihout extra dependencies
  ☆14Updated 8 months ago
• saracen / llama2.go
  Inference Llama 2 in Go
  ☆39Updated last year
• harness / harness-migrate
  Harness CI migration utility
  ☆11Updated this week
• zincsearch / wal
  Write ahead log for Go.
  ☆10Updated last year
• getzep / zep-go
  Zep: Long-Term Memory for ‍AI Assistants (Go SDK)
  ☆23Updated last week
• gomlx / gemma
  GoMLX implementation of Google's Gemma models
  ☆25Updated 8 months ago
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆42Updated 5 months ago
• robbyt / llm_proxy
  AI aware proxy
  ☆19Updated 10 months ago
• mariotoffia / goannoy
  go native port of annoy. Approximate Nearest Neighbors in optimized for memory usage and loading/saving to disk.
  ☆18Updated 7 months ago
• eliben / go-sentencepiece
  Go implementation of the SentencePiece tokenizer
  ☆31Updated 10 months ago
• dwisiswant0 / fastdns
  DNS at ludicrous speed for Go, powered by XDP sockets. [EXPERIMENTAL]
  ☆11Updated 3 weeks ago
• mudler / go-stable-diffusion
  ☆16Updated last year
• blast-go / openblas
  OpenBLAS binding for go
  ☆13Updated 2 years ago
• mikestefanello / batcher
  Type-safe, automatic, asynchronous batch processing.
  ☆18Updated last year
• igolaizola / igogpt
  AutoGPT and other GPT and AI tools written in Golang
  ☆73Updated last year
• struct-chat / embedding
  Vector Embedding Server in under 100 lines of code
  ☆22Updated last year
• antirez / LLM-FTC-sampling
  First token cutoff sampling inference example
  ☆30Updated last year
• stillmatic / gollum
  Production grade LLM-ops in Golang
  ☆55Updated this week
• tailscale / goexpect
  Expect for Go
  ☆11Updated last year
• gomlx / gopjrt
  Go Wrappers for OpenXLA PJRT
  ☆28Updated last month
• cornelk / llama-go
  Port of Facebook's LLaMA (Large Language Model Meta AI) in Golang with embedded C/C++
  ☆168Updated 2 years ago
• fogfish / hnsw
  Hierarchical Navigable Small World Graphs
  ☆12Updated 11 months ago
• gomlx / go-huggingface
  Conveniently download files, models, tokenizers from HuggingFace Hub
  ☆31Updated 3 weeks ago
• robbyt / go-supervisor
  Handle OS signals, graceful shutdown, hot reloading
  ☆29Updated 3 weeks ago