Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• go-skynet / go-ggml-transformers.cpp
  Binding to transformers in ggml
  ☆64Updated last week
• gotzmann / booster
  Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
  ☆165Updated last year
• getzep / zep-go
  Build Agents That Recall What Matters. Systematically engineer relevant context from chat history & business data. (Go SDK)
  ☆36Updated 3 weeks ago
• pjbgf / sha1cd
  SHA1 implementation with collision detection.
  ☆14Updated last week
• shoenig / go-landlock
  Go filesystem isolation via Linux landlock sandbox
  ☆22Updated 9 months ago
• stillmatic / gollum
  Production grade LLM-ops in Golang
  ☆57Updated last week
• kardianos / mitmproxy
  ☆26Updated last year
• almogtavor / facts-checker-gpt
  Perform facts checks on your conversations with LLMs to catch fake-news, misleading information, and LLMs confusion.
  ☆12Updated 2 years ago
• eliben / go-sentencepiece
  Go implementation of the SentencePiece tokenizer
  ☆38Updated last week
• lrstanley / go-queryparser
  "q" http GET variable parser that supports filters/tags for advanced searches
  ☆15Updated 2 months ago
• milosgajdos / embeviz
  A simple app for visualising text embeddings
  ☆23Updated 5 months ago
• tmc / go-llama2
  Llama 2 inference in one file of pure Go
  ☆109Updated 2 years ago
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆42Updated 10 months ago
• mudler / go-stable-diffusion
  ☆17Updated last year
• alecthomas / atomic
  Type-safe atomic values for Go
  ☆19Updated 2 years ago
• c0mm4nd / wasman
  Another wasm interpreter engine for gophers based on pure golang
  ☆11Updated 3 years ago
• Kurt212 / syncgroup
  Golang goroutines synchronization primitive wihout extra dependencies
  ☆14Updated last year
• saracen / llama2.go
  Inference Llama 2 in Go
  ☆39Updated 2 years ago
• anacrolix / mmsg
  Exposes batch message receives (recvmmsg)
  ☆14Updated 4 months ago
• fogfish / word2vec
  Golang "native" implementation of word2vec algorithm (word2vec++ port)
  ☆30Updated 2 months ago
• cornelk / llama-go
  Port of Facebook's LLaMA (Large Language Model Meta AI) in Golang with embedded C/C++
  ☆169Updated 2 years ago
• acheong08 / vectordb
  A simple vector database: Text encoding, semantic search, document storage
  ☆90Updated 2 years ago
• neuml / txtai.go
  Go client for txtai
  ☆80Updated 3 weeks ago
• riobard / go-mmap
  Easy handling of memory-mapped files
  ☆22Updated 11 years ago
• camdencheek / simd_blog
  ☆19Updated 2 years ago
• zenmodel / zenmodel
  ZenModel is a framework for building LLM applications with agentic workflow
  ☆74Updated last year
• haormj / llama2.go
  Inference Llama 2 in one file of pure go
  ☆16Updated 2 years ago
• fbiville / markdown-table-formatter
  Markdown table formatter in Go
  ☆20Updated 4 years ago
• go-nlp / tfidf
  tfidf provides TF-IDF functionality
  ☆13Updated 2 years ago
• gokrazy / wifi
  WiFi daemon for connecting to WiFi networks (encrypted or unencrypted) with gokrazy
  ☆37Updated 7 months ago