javirandor / anthropic-tokenizer
Approximation of the Claude 3 tokenizer by inspecting generation stream
☆116Updated 4 months ago
Related projects ⓘ
Alternatives and complementary repositories for anthropic-tokenizer
- Just a bunch of benchmark logs for different LLMs☆116Updated 3 months ago
- Steer LLM outputs towards a certain topic/subject and enhance response capabilities using activation engineering by adding steering vecto…☆204Updated 6 months ago
- Comprehensive analysis of difference in performance of QLora, Lora, and Full Finetunes.☆81Updated last year
- Sphynx Hallucination Induction☆48Updated 3 months ago
- MiniHF is an inference, human preference data collection, and fine-tuning tool for local language models. It is intended to help the user…☆152Updated last week
- Doing simple retrieval from LLM models at various context lengths to measure accuracy☆97Updated 7 months ago
- ☆104Updated 8 months ago
- ☆72Updated last year
- Red-Teaming Language Models with DSPy☆142Updated 7 months ago
- ☆48Updated last year
- This repository explains and provides examples for "concept anchoring" in GPT4.☆72Updated 10 months ago
- Simple embedding -> text model trained on a small subset of Wikipedia sentences.☆152Updated last year
- smolLM with Entropix sampler on pytorch☆141Updated 3 weeks ago
- an implementation of Self-Extend, to expand the context window via grouped attention☆118Updated 10 months ago
- Generate Synthetic Data Using OpenAI, MistralAI or AnthropicAI☆221Updated 6 months ago
- Aidan Bench attempts to measure <big_model_smell> in LLMs.☆100Updated this week
- An easy-to-understand framework for LLM samplers that rewind and revise generated tokens☆113Updated 3 weeks ago
- auto fine tune of models with synthetic data☆72Updated 9 months ago
- Track the progress of LLM context utilisation☆53Updated 4 months ago
- A simple Python sandbox for helpful LLM data agents☆173Updated 5 months ago
- ☆101Updated 3 months ago
- Mixing Language Models with Self-Verification and Meta-Verification☆97Updated last year
- Reimplementation of the task generation part from the Alpaca paper☆119Updated last year
- ☆150Updated 4 months ago
- Attribute (or cite) statements generated by LLMs back to in-context information.☆148Updated last month
- look how they massacred my boy☆58Updated last month
- ☆64Updated 5 months ago
- This open-source repository offers reference code for integrating workplace datastores with Cohere's LLMs, enabling developers and busine…☆143Updated last month
- 📝 Reference-Free automatic summarization evaluation with potential hallucination detection☆98Updated 10 months ago