Alternatives and similar repositories for institutional-books-1-pipeline

Users that are interested in institutional-books-1-pipeline are comparing it to the libraries listed below

• r-three / common-pile
  Code for collecting, processing, and preparing datasets for the Common Pile
  ☆248Updated 3 months ago
• EleutherAI / tokengrams
  Efficiently computing & storing token n-grams from large corpora
  ☆26Updated last year
• Pleias / Pleias-RAG-Library
  Python library to use Pleias-RAG models
  ☆67Updated 7 months ago
• jxmorris12 / embzip
  lossily compress representation vectors using product quantization
  ☆59Updated last month
• antirez / LLM-FTC-sampling
  First token cutoff sampling inference example
  ☆31Updated last year
• dropbox / aana_sdk
  Aana SDK is a powerful framework for building AI enabled multimodal applications.
  ☆55Updated 4 months ago
• furiousteabag / vram-calculator
  Transformer GPU VRAM estimator
  ☆67Updated last year
• METR / eval-analysis-public
  Public repository containing METR's DVC pipeline for eval data analysis
  ☆164Updated 8 months ago
• codelion / pts
  Pivotal Token Search
  ☆141Updated last week
• kyegomez / Exa
  Unleash the full potential of exascale LLMs on consumer-class GPUs, proven by extensive benchmarks, with no long-term adjustments and min…
  ☆26Updated last year
• foundation-model-stack / bamba
  Train, tune, and infer Bamba model
  ☆137Updated 6 months ago
• simonw / llm-cluster
  LLM plugin for clustering embeddings
  ☆82Updated last year
• simonw / llm-embed-jina
  Embedding models from Jina AI
  ☆65Updated last year
• Pleias / OCRoscope
  Small python package to measure OCR quality and other related metrics.
  ☆25Updated last year
• LLM360 / crystalcoder-train
  Pre-training code for CrystalCoder 7B LLM
  ☆55Updated last year
• dwarvesf / llm-hosting
  This repository is designed for deploying and managing server processes that handle embeddings using the Infinity Embedding model or Larg…
  ☆26Updated 9 months ago
• Pleias / Quest-Best-Tokens
  An introduction to LLM Sampling
  ☆79Updated last year
• cfahlgren1 / hf-data-explorer
  Chrome Extension for exploring Hugging Face datasets 🔎
  ☆49Updated last year
• Nicolas-BZRD / EuroBERT
  Optimus is a flexible and scalable framework built to train language models efficiently across diverse hardware configurations, including…
  ☆67Updated 3 weeks ago
• cgftinc / benchmax
  Framework-Agnostic RL Environments for LLM Fine-Tuning
  ☆40Updated 3 weeks ago
• IBM / text-generation-inference
  IBM development fork of https://github.com/huggingface/text-generation-inference
  ☆62Updated 3 months ago
• allenai / duplodocus
  Tooling for exact and MinHash deduplication of large-scale text datasets
  ☆46Updated last week
• ibm-granite / granite-3.1-language-models
  Granite 3.1 Language Models
  ☆135Updated 6 months ago
• enjalot / latent-sae
  Training code for Sparse Autoencoders on Embedding models
  ☆39Updated 10 months ago
• simonw / llm-nomic-api-embed
  Create embeddings for LLM using the Nomic API
  ☆23Updated last year
• teknium1 / LLM-Logbook
  Public reports detailing responses to sets of prompts by Large Language Models.
  ☆32Updated 11 months ago
• babylm / baseline-pretraining
  Code for pre-training BabyLM baseline models.
  ☆16Updated 2 years ago
• JHU-CLSP / mmBERT
  A massively multilingual modern encoder language model
  ☆117Updated 2 months ago
• reka-ai / rekaquant
  ☆62Updated 5 months ago
• codestoryai / swe_bench_traces
  Contains the model patches and the eval logs from the passing swe-bench-lite run.
  ☆10Updated last year