tiktoken-go / tokenizer
Pure Go implementation of OpenAI's tiktoken tokenizer
☆330Updated this week
Alternatives and similar repositories for tokenizer:
Users that are interested in tokenizer are comparing it to the libraries listed below
- pgvector support for Go☆217Updated this week
- NLP tokenizers written in Go language☆216Updated 3 months ago
- Fully featured, spec-compliant HTML5 server-sent events library☆385Updated last month
- The Go client for Chroma vector database☆103Updated 3 weeks ago
- Package tasks is an easy to use in-process scheduler for recurring tasks in Go☆309Updated 6 months ago
- 🚨 Design workflows of slog handlers: pipeline, middleware, fanout, routing, failover, load balancing...☆450Updated last month
- go-mask is a simple, customizable Go library for masking sensitive information.☆190Updated 7 months ago
- 🚨 slog: Attribute formatting☆161Updated last month
- Cybertron: the home planet of the Transformers in Go☆299Updated 9 months ago
- Valgo is a type-safe, expressive, and extensible validator library for Golang. Supports localization and is built with generics.☆444Updated 9 months ago
- ☆158Updated 8 months ago
- LLaMA-2 in native Go☆191Updated 3 months ago
- Anthropic Claude API wrapper for Go☆135Updated last week
- go version of tiktoken☆723Updated 9 months ago
- Go client for Qdrant vector search engine☆192Updated last month
- Is a general library to help modify or set data within data structures and other objects.☆255Updated 2 months ago
- Go client libraries for OpenAI☆452Updated last year
- Copygen generates code based on Go types. Generate type-based code to copy values from type to type and fields from struct to struct by d…☆373Updated last week
- Onnx transformer pipelines in Golang☆363Updated this week
- Maths includes mathematical functions not defined in the standard Go math package.☆174Updated last year
- Go slog's Handler implementation to ensure correlation with Open-Telemetry spans.☆123Updated last month
- NLP transformers written in Go☆222Updated 2 years ago
- A minimalist Go configuration library☆381Updated last year
- Fast, high quality alternative to math/rand and golang.org/x/exp/rand☆239Updated 4 months ago
- 🪿 LinGoose is a Go framework for building awesome AI/LLM applications.☆732Updated this week
- Low-level Go interface to SQLite 3☆853Updated this week
- Caches Dialector☆117Updated 10 months ago
- A zero-dependency generic in-memory cache Go library☆123Updated 2 months ago
- Go implementation to calculate Levenshtein Distance.☆381Updated last month
- Go bindings for HuggingFace Tokenizer☆117Updated 4 months ago