Alternatives and similar repositories for tokenizer

Users that are interested in tokenizer are comparing it to the libraries listed below

• go-skynet / go-ggml-transformers.cpp
  Binding to transformers in ggml
  ☆64Updated last week
• milosgajdos / embeviz
  A simple app for visualising text embeddings
  ☆23Updated 6 months ago
• gotzmann / booster
  Booster - open accelerator for LLM models. Better inference and debugging for AI hackers
  ☆167Updated last year
• c0mm4nd / wasman
  Another wasm interpreter engine for gophers based on pure golang
  ☆11Updated 3 years ago
• mmatongo / chew
  Chew is a Go library for processing various content types into markdown/plaintext.
  ☆42Updated 11 months ago
• shoenig / go-landlock
  Go filesystem isolation via Linux landlock sandbox
  ☆22Updated 11 months ago
• anacrolix / mmsg
  Exposes batch message receives (recvmmsg)
  ☆14Updated 5 months ago
• openpcc / bhttp
  BHTTP - Binary HTTP Message Format in golang
  ☆25Updated last month
• kardianos / mitmproxy
  ☆26Updated last year
• CyberChainXyz / go-opencl
  Run OpenCL programs in Go programs easily.
  ☆17Updated last year
• eliben / go-sentencepiece
  Go implementation of the SentencePiece tokenizer
  ☆46Updated last month
• harness / harness-migrate
  Harness CI migration utility
  ☆11Updated last month
• Broderick-Westrope / amalgo
  CLI tool for creating single-file snapshots of multiple (code) files.
  ☆17Updated last year
• Kurt212 / syncgroup
  Golang goroutines synchronization primitive wihout extra dependencies
  ☆14Updated last year
• alecthomas / atomic
  Type-safe atomic values for Go
  ☆19Updated 2 years ago
• atomicgo / ghissue
  📦 Go library that enables your users to submit error reports to GitHub
  ☆13Updated last year
• stillmatic / gollum
  Production grade LLM-ops in Golang
  ☆58Updated this week
• gomlx / go-huggingface
  Conveniently download files, models, tokenizers from HuggingFace Hub
  ☆49Updated 2 weeks ago
• camdencheek / simd_blog
  ☆19Updated 2 years ago
• goki / vulkan
  Vulkan API bindings for Go programming language
  ☆10Updated last year
• lrstanley / go-queryparser
  "q" http GET variable parser that supports filters/tags for advanced searches
  ☆15Updated 3 months ago
• lucyknada / detective-needle-llm
  ☆12Updated last year
• firecrawl / firecrawl-go
  ☆26Updated 4 months ago
• tmc / go-llama2
  Llama 2 inference in one file of pure Go
  ☆111Updated 2 years ago
• beevik / nts
  network time security client package for go
  ☆19Updated 3 weeks ago
• saracen / llama2.go
  Inference Llama 2 in Go
  ☆39Updated 2 years ago
• tklauser / ps
  List, find and inspect operating system processes in Go
  ☆27Updated 2 weeks ago
• llmcontext / gomcp
  Unofficial Golang SDK for Anthropic Model Context Protocol
  ☆53Updated last year
• sunfish-shogi / bufseekio
  ☆11Updated 3 years ago
• yuseferi / envyaml
  env Yaml is a configuration loader from Yaml file with enviromental variable that helps to have secret placeholders in yaml file.
  ☆17Updated last month