Alternatives and similar repositories for autotiktokenizer:

Users that are interested in autotiktokenizer are comparing it to the libraries listed below

• ali-bahrainian / RAG_best_practices
  ☆58Updated last week
• vultureprime / deploy-ai-model
  ☆24Updated last year
• louisbrulenaudet / ragoon
  High level library for batched embeddings generation, blazingly-fast web-based RAG and quantized indexes processing ⚡
  ☆63Updated 2 months ago
• PyThaiNLP / WangChanGLM
  WangChanGLM 🐘 - The Multilingual Instruction-Following Model
  ☆94Updated last year
• Locutusque / TPU-Alignment
  Fully fine-tune large models like Mistral, Llama-2-13B, or Qwen-14B completely for free
  ☆224Updated 2 months ago
• Pleias / Various-Finetuning
  Set of scripts to finetune LLMs
  ☆36Updated 9 months ago
• vistec-AI / WangchanX
  WangchanX Fine-tuning Pipeline
  ☆44Updated 3 months ago
• anyscale / e2e-llm-workflows
  End-to-End LLM Guide
  ☆100Updated 6 months ago
• premAI-io / benchmarks
  🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.
  ☆138Updated 5 months ago
• PyThaiNLP / KhanomTanLLM
  ☆12Updated 3 months ago
• huggingface / optimum-tpu
  Google TPU optimizations for transformers models
  ☆88Updated this week
• huggingface / competitions
  ☆117Updated 2 months ago
• zjohn77 / lightning-mlflow-hf
  Use QLoRA to tune LLM in PyTorch-Lightning w/ Huggingface + MLflow
  ☆57Updated last year
• AlexBodner / How_Much_VRAM
  ☆96Updated 4 months ago
• AnswerDotAI / toolslm
  Tools to make language models a bit easier to use
  ☆33Updated this week
• nahidalam / maya
  Maya: An Instruction Finetuned Multilingual Multimodal Model using Aya
  ☆100Updated 2 weeks ago
• weaviate / structured-rag
  Experimental Code for StructuredRAG: JSON Response Formatting with Large Language Models
  ☆99Updated last month
• aniketmaurya / python-project-template
  A template to kick-start your Python project ✨🚀
  ☆51Updated 3 weeks ago
• jxmorris12 / cde
  code for training & evaluating Contextual Document Embedding models
  ☆166Updated last week
• NielsRogge / awesome-huggingface
  Repository containing awesome resources regarding Hugging Face tooling.
  ☆46Updated last year
• mixedbread-ai / batched
  ☆119Updated last month
• PrithivirajDamodaran / SPLADERunner
  Lite weight wrapper for the independent implementation of SPLADE++ models for search & retrieval pipelines. Models and Library created by…
  ☆29Updated 4 months ago
• S1M0N38 / dspy-arxiv
  Explore the use of DSPy for extracting features from PDFs 🔎
  ☆38Updated 10 months ago
• IlyasMoutawwakil / py-txi
  A Python wrapper around HuggingFace's TGI (text-generation-inference) and TEI (text-embedding-inference) servers.
  ☆34Updated last month
• cfahlgren1 / observers
  A Lightweight Library for AI Observability
  ☆230Updated this week
• huggingface / llm-swarm
  Manage scalable open LLM inference endpoints in Slurm clusters
  ☆248Updated 6 months ago
• deshwalmahesh / PHUDGE
  Official repo for the paper PHUDGE: Phi-3 as Scalable Judge. Evaluate your LLMs with or without custom rubric, reference answer, absolute…
  ☆48Updated 6 months ago
• joey00072 / ohara
  Collection of autoregressive model implementation
  ☆76Updated 2 weeks ago
• hamelsmu / llama-inference
  experiments with inference on llama
  ☆104Updated 7 months ago
• ariG23498 / quantized-diffusion-inference
  Notebook and Scripts that showcase running quantized diffusion models on consumer GPUs
  ☆37Updated 2 months ago