Alternatives and similar repositories for gpt4-tokenizer

Users that are interested in gpt4-tokenizer are comparing it to the libraries listed below

• benlipkin / probsem
  Probabilistic LLM evaluations. [CogSci2023; ACL2023]
  ☆73Updated 9 months ago
• stunningpixels / lou-eval
  Track the progress of LLM context utilisation
  ☆54Updated last month
• ruiqi-zhong / D5
  The GitHub repo for Goal Driven Discovery of Distributional Differences via Language Descriptions
  ☆69Updated 2 years ago
• mobarski / alpaca-libre
  Reimplementation of the task generation part from the Alpaca paper
  ☆119Updated 2 years ago
• teknium1 / transformers-gptq-quant
  ☆48Updated last year
• InflectionAI / Inflection-Benchmarks
  Public Inflection Benchmarks
  ☆68Updated last year
• scottlogic-alex / prm800k-denorm
  Script for processing OpenAI's PRM800K process supervision dataset into an Alpaca-style instruction-response format
  ☆27Updated last year
• qrdlgit / graph-of-thoughts
  Based on the tree of thoughts paper
  ☆48Updated last year
• allenai / CommonGen-Eval
  Evaluating LLMs with CommonGen-Lite
  ☆90Updated last year
• Alignment-Lab-AI / datagen
  a pipeline for using api calls to agnostically convert unstructured data into structured training data
  ☆30Updated 7 months ago
• curai / curai-research
  ☆94Updated 4 months ago
• CarperAI / treasure_trove
  ☆22Updated last year
• laramohan / wikillm
  LLMs as Collaboratively Edited Knowledge Bases
  ☆45Updated last year
• EleutherAI / semantic-memorization
  ☆44Updated 6 months ago
• Edward-Sun / RECITE
  Code of ICLR paper: https://openreview.net/forum?id=-cqvvvb-NkI
  ☆94Updated 2 years ago
• CG80499 / trlx-with-T5
  [Added T5 support to TRLX] A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)
  ☆47Updated 2 years ago
• CarperAI / decontamination
  This repository contains code for cleaning your training data of benchmark data to help combat data snooping.
  ☆25Updated 2 years ago
• shisa-ai / shisa-v2
  Japanese / English Bilingual LLM
  ☆15Updated last week
• jlin816 / dialop
  DialOp: Decision-oriented dialogue environments for collaborative language agents
  ☆106Updated 6 months ago
• stanfordnlp / multi-distribution-retrieval
  Code for our paper Resources and Evaluations for Multi-Distribution Dense Information Retrieval
  ☆14Updated last year
• CarperAI / InstructGPT
  For experiments involving instruct gpt. Currently used for documenting open research questions.
  ☆71Updated 2 years ago
• mungg / FABLES
  ☆57Updated 7 months ago
• knowrohit / know_medical_dialogues
  KMD is a collection of conversational exchanges between patients and doctors on various medical topics. It aims to capture the intricaci…
  ☆24Updated last year
• schen149 / sub-sentence-encoder
  The official code repo for "Sub-Sentence Encoder: Contrastive Learning of Propositional Semantic Representations".
  ☆82Updated last year
• teknium1 / LLM-Benchmark-Logs
  Just a bunch of benchmark logs for different LLMs
  ☆118Updated 9 months ago
• google-research-datasets / QAmeleon
  QAmeleon introduces synthetic multilingual QA data using PaLM, a 540B large language model. This dataset was generated by prompt tuning P…
  ☆34Updated last year
• KaiNylund / lm-weights-encode-time
  ☆68Updated 9 months ago
• allenai / infinigram-api
  ☆57Updated this week
• shrahimim / Boosting-Theory-of-Mind-in-LLMs-with-Prompting
  ☆32Updated 2 years ago
• kyegomez / Kosmos-X
  The Next Generation Multi-Modality Superintelligence
  ☆71Updated 8 months ago