Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• go-skynet / go-ggml-transformers.cpp
  Binding to transformers in ggml
  ☆63Updated this week
• gotzmann / booster
  Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
  ☆162Updated last year
• Kurt212 / syncgroup
  Golang goroutines synchronization primitive wihout extra dependencies
  ☆14Updated 10 months ago
• c0mm4nd / wasman
  Another wasm interpreter engine for gophers based on pure golang
  ☆11Updated 3 years ago
• pjbgf / sha1cd
  SHA1 implementation with collision detection.
  ☆14Updated this week
• almogtavor / facts-checker-gpt
  Perform facts checks on your conversations with LLMs to catch fake-news, misleading information, and LLMs confusion.
  ☆12Updated 2 years ago
• saracen / llama2.go
  Inference Llama 2 in Go
  ☆39Updated 2 years ago
• milosgajdos / embeviz
  A simple app for visualising text embeddings
  ☆22Updated last month
• matthewrennie / go-llama.cpp
  Go bindings for LLama.cpp
  ☆13Updated 2 years ago
• yalue / onnxruntime_go_examples
  Example applications using the onnxruntime_go library.
  ☆27Updated 2 months ago
• kardianos / mitmproxy
  ☆26Updated last year
• anacrolix / mmsg
  Exposes batch message receives (recvmmsg)
  ☆13Updated 3 weeks ago
• sunfish-shogi / bufseekio
  ☆11Updated 3 years ago
• mudler / go-stable-diffusion
  ☆16Updated last year
• haormj / llama2.go
  Inference Llama 2 in one file of pure go
  ☆16Updated 2 years ago
• XiaoConstantine / dspy-go
  DSPy Go implementation
  ☆89Updated 2 weeks ago
• stillmatic / gollum
  Production grade LLM-ops in Golang
  ☆57Updated this week
• tmc / go-llama2
  Llama 2 inference in one file of pure Go
  ☆106Updated 2 years ago
• mikestefanello / batcher
  Type-safe, automatic, asynchronous batch processing.
  ☆18Updated last year
• alecthomas / atomic
  Type-safe atomic values for Go
  ☆19Updated last year
• acheong08 / vectordb
  A simple vector database: Text encoding, semantic search, document storage
  ☆90Updated 2 years ago
• zenmodel / zenmodel
  ZenModel is a framework for building LLM applications with agentic workflow
  ☆72Updated 10 months ago
• lucyknada / detective-needle-llm
  ☆12Updated 11 months ago
• donomii / go-rwkv.cpp
  A go wrapper around the rwkv.cpp library
  ☆20Updated last year
• harness / harness-migrate
  Harness CI migration utility
  ☆10Updated this week
• eliben / go-sentencepiece
  Go implementation of the SentencePiece tokenizer
  ☆33Updated last year
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆42Updated 6 months ago
• cornelk / llama-go
  Port of Facebook's LLaMA (Large Language Model Meta AI) in Golang with embedded C/C++
  ☆168Updated 2 years ago
• fogfish / word2vec
  Golang "native" implementation of word2vec algorithm (word2vec++ port)
  ☆28Updated 8 months ago
• blast-go / openblas
  OpenBLAS binding for go
  ☆13Updated 2 years ago