Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• go-skynet / go-ggml-transformers.cpp
  Binding to transformers in ggml
  ☆64Updated last week
• gotzmann / booster
  Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
  ☆167Updated last year
• milosgajdos / embeviz
  A simple app for visualising text embeddings
  ☆23Updated 6 months ago
• Kurt212 / syncgroup
  Golang goroutines synchronization primitive wihout extra dependencies
  ☆14Updated last year
• shoenig / go-landlock
  Go filesystem isolation via Linux landlock sandbox
  ☆22Updated 11 months ago
• sunfish-shogi / bufseekio
  ☆11Updated 3 years ago
• almogtavor / facts-checker-gpt
  Perform facts checks on your conversations with LLMs to catch fake-news, misleading information, and LLMs confusion.
  ☆12Updated 2 years ago
• lrstanley / go-queryparser
  "q" http GET variable parser that supports filters/tags for advanced searches
  ☆15Updated 3 months ago
• anacrolix / mmsg
  Exposes batch message receives (recvmmsg)
  ☆14Updated 5 months ago
• stillmatic / gollum
  Production grade LLM-ops in Golang
  ☆58Updated this week
• kardianos / mitmproxy
  ☆26Updated last year
• c0mm4nd / wasman
  Another wasm interpreter engine for gophers based on pure golang
  ☆11Updated 3 years ago
• getzep / zep-go
  Build Agents That Recall What Matters. Systematically engineer relevant context from chat history & business data. (Go SDK)
  ☆43Updated 2 weeks ago
• alecthomas / atomic
  Type-safe atomic values for Go
  ☆19Updated 2 years ago
• matthewrennie / go-llama.cpp
  Go bindings for LLama.cpp
  ☆14Updated 2 years ago
• dalzilio / rudd
  Rudd is a Binary Decision Diagram (BDD) library written in pure Go.
  ☆13Updated last year
• eliben / go-sentencepiece
  Go implementation of the SentencePiece tokenizer
  ☆46Updated last month
• gomlx / go-huggingface
  Conveniently download files, models, tokenizers from HuggingFace Hub
  ☆49Updated 2 weeks ago
• zenmodel / zenmodel
  ZenModel is a framework for building LLM applications with agentic workflow
  ☆74Updated last year
• fbiville / markdown-table-formatter
  Markdown table formatter in Go
  ☆20Updated 4 years ago
• gptscript-ai / go-gptscript
  Go module for running GPTScript
  ☆12Updated this week
• pjbgf / sha1cd
  SHA1 implementation with collision detection.
  ☆15Updated 3 weeks ago
• lucyknada / detective-needle-llm
  ☆12Updated last year
• openpcc / bhttp
  BHTTP - Binary HTTP Message Format in golang
  ☆25Updated last month
• blast-go / openblas
  OpenBLAS binding for go
  ☆14Updated 2 years ago
• igolaizola / igogpt
  AutoGPT and other GPT and AI tools written in Golang
  ☆72Updated last year
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆42Updated 11 months ago
• saracen / llama2.go
  Inference Llama 2 in Go
  ☆39Updated 2 years ago
• atomicgo / ghissue
  📦 Go library that enables your users to submit error reports to GitHub
  ☆13Updated last year
• cloudwego / base64x
  High performance drop-in replacement of the encoding/base64 library.
  ☆21Updated 6 months ago