dqbd / tiktokenizer
Online playground for OpenAPI tokenizers
☆1,070Updated last month
Alternatives and similar repositories for tiktokenizer:
Users that are interested in tiktokenizer are comparing it to the libraries listed below
- Inference engine powering open source models on OpenRouter☆814Updated 2 months ago
- HTML to Markdown converter and crawler.☆532Updated last year
- Python & JS/TS SDK for running AI-generated code/code interpreting in your AI app☆1,613Updated this week
- Run Mixtral-8x7B models in Colab or consumer desktops☆2,302Updated 11 months ago
- MTEB: Massive Text Embedding Benchmark☆2,350Updated this week
- Cohere Toolkit is a collection of prebuilt components enabling users to quickly build and deploy RAG applications.☆3,014Updated 2 weeks ago
- Data and tools for generating and inspecting OLMo pre-training data.☆1,173Updated 2 weeks ago
- Doing simple retrieval from LLM models at various context lengths to measure accuracy☆1,778Updated 7 months ago
- Common interface for interacting with AI agents. The protocol is tech stack agnostic - you can use it with any framework for building age…☆1,151Updated 2 months ago
- JS tokenizer for LLaMA 1 and 2☆351Updated 9 months ago
- ☆451Updated last year
- A tiny library for coding with large language models.☆1,227Updated 8 months ago
- 🍰 PromptLayer - Maintain a log of your prompts and OpenAI API requests. Track, debug, and replay old completions.☆567Updated this week
- Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verifi…☆2,601Updated last week
- ☆438Updated 5 months ago
- ☆842Updated 6 months ago
- 📊 llm.report is an open-source logging and analytics platform for OpenAI: Log your ChatGPT API requests, analyze costs, and improve your…☆995Updated 10 months ago
- Chat with AI large language models running natively in your browser. Enjoy private, server-free, seamless AI conversations.☆697Updated 2 months ago
- ☆763Updated last year
- A python wrapper for Tavily search API☆580Updated this week
- Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024☆1,418Updated 2 months ago
- Evaluation tool for LLM QA chains☆1,072Updated last year
- Analyzing the evolution of ChatGPT's codebase through time with curated archives and scripts☆280Updated 7 months ago
- A library of data loaders for LLMs made by the community -- to be used with LlamaIndex and/or LangChain☆3,474Updated last year
- LangChain-powered web researcher chatbot. Searches for sources on the web and cites them in generated answers.☆540Updated last year
- LangServe 🦜️🏓☆2,051Updated 3 months ago
- LLMPerf is a library for validating and benchmarking LLMs☆845Updated 3 months ago
- ☆695Updated last month
- Examples of using E2B☆900Updated this week
- Build, customize and control you own LLMs. From data pre-processing to fine-tuning, xTuring provides an easy way to personalize open-sour…☆2,640Updated 6 months ago