leogao2/lm_dataformat external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

leogao2/lm_dataformat

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/leogao2/lm_dataformat)

Close

leogao2 / lm_dataformatView on GitHubMore

• External links
• Readme badge

☆78Dec 7, 2023Updated 2 years ago

Alternatives and similar repositories for lm_dataformat

Users that are interested in lm_dataformat are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• EleutherAI / openwebtext2

  View on GitHub
  ☆95Jul 16, 2022Updated 3 years ago
• monologg / ko_lm_dataformat

  View on GitHub
  A utility for storing and reading files for Korean LM training 💾
  ☆35Oct 15, 2025Updated 7 months ago
• EleutherAI / the-pile

  View on GitHub
  ☆1,661Apr 27, 2023Updated 3 years ago
• EleutherAI / github-downloader

  View on GitHub
  Script for downloading GitHub.
  ☆99Jul 1, 2024Updated last year
• EleutherAI / pile_dedupe

  View on GitHub
  Pile Deduplication Code
  ☆18May 15, 2023Updated 3 years ago
• AI Agents on DigitalOcean Gradient AI Platform • Ad
  Build production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
• EleutherAI / project-menu

  View on GitHub
  See the issue board for the current status of active and prospective projects!
  ☆65Feb 12, 2022Updated 4 years ago
• tlkh / t2t-tuner

  View on GitHub
  Convenient Text-to-Text Training for Transformers
  ☆18Dec 10, 2021Updated 4 years ago
• noanabeshima / wikipedia-downloader

  View on GitHub
  Downloads 2020 English Wikipedia articles as plaintext
  ☆27Mar 25, 2023Updated 3 years ago
• EleutherAI / DeeperSpeed

  View on GitHub
  DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
  ☆173Sep 26, 2025Updated 8 months ago
• fastai / fastpy

  View on GitHub
  An easy way to start a python programming environment using GitHub Codespaces.
  ☆15Sep 9, 2020Updated 5 years ago
• AnswerDotAI / nbdev-index

  View on GitHub
  nbdev docs lookup for a few libraries and python itself
  ☆25Feb 9, 2026Updated 4 months ago
• zphang / minimal-gpt-neox-20b

  View on GitHub
  ☆131Jun 9, 2022Updated 4 years ago
• hpcaitech / ColossalAI-Pytorch-lightning

  View on GitHub
  ☆24Nov 22, 2022Updated 3 years ago
• mattf1n / basis-aware-threshold

  View on GitHub
  Code for the paper "Closing the Curious Case of Neural Text Degeneration"
  ☆12Apr 9, 2025Updated last year
• Deploy to Railway using AI coding agents - Free Credits Offer • Ad
  Use Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
• EleutherAI / pyfra

  View on GitHub
  Python Research Framework
  ☆107Nov 3, 2022Updated 3 years ago
• AIRC-KETI / Korean-Copora

  View on GitHub
  ☆14Dec 9, 2021Updated 4 years ago
• jason9693 / oslo-kogpt-finetunig

  View on GitHub
  kogpt를 oslo로 파인튜닝하는 예제.
  ☆23Aug 26, 2022Updated 3 years ago
• jenni-ai / T2FW

  View on GitHub
  Fine-Tuning Pre-trained Transformers into Decaying Fast Weights
  ☆19Oct 9, 2022Updated 3 years ago
• AranKomat / Metroplex

  View on GitHub
  ☆21Mar 15, 2023Updated 3 years ago
• nlpai-lab / Korean-CommonGen

  View on GitHub
  [Findings of NAACL2022] A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation
  ☆11May 27, 2022Updated 4 years ago
• EleutherAI / semantic-memorization

  View on GitHub
  ☆44Nov 17, 2024Updated last year
• sdtblck / youtube_subtitle_dataset

  View on GitHub
  YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training
  ☆46Sep 22, 2020Updated 5 years ago
• Beomi / transformers-language-modeling

  View on GitHub
  Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3
  ☆23May 20, 2021Updated 5 years ago
• Deploy to Railway using AI coding agents - Free Credits Offer • Ad
  Use Claude Code, Codex, OpenCode, and more. Autonomous software development now has the infrastructure to match with Railway.
• XinyuHua / textgen-emnlp19

  View on GitHub
  Code for our EMNLP 2019 paper titled "Sentence-Level Content Planning and Style Specification for Neural Text Generation"
  ☆17May 4, 2020Updated 6 years ago
• lucidrains / all-normalization-transformer

  View on GitHub
  A simple Transformer where the softmax has been replaced with normalization
  ☆20Sep 11, 2020Updated 5 years ago
• zphang / minimal-opt

  View on GitHub
  ☆67Aug 24, 2022Updated 3 years ago
• fastai / docments

  View on GitHub
  Document parameters using comments
  ☆10Aug 6, 2021Updated 4 years ago
• EleutherAI / tokengrams

  View on GitHub
  Efficiently computing & storing token n-grams from large corpora
  ☆27Oct 6, 2024Updated last year
• lucidrains / rela-transformer

  View on GitHub
  Implementation of a Transformer using ReLA (Rectified Linear Attention) from https://arxiv.org/abs/2104.07012
  ☆49Apr 6, 2022Updated 4 years ago
• LIJUNYI95 / SuperAdam

  View on GitHub
  Official Pytorch Implementation for the paper 'SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients'
  ☆17Jan 12, 2022Updated 4 years ago
• AIRC-KETI / ke-t5-downstreams

  View on GitHub
  ☆39Mar 25, 2024Updated 2 years ago
• AeroScripts / HiddenEngrams

  View on GitHub
  Hidden Engrams: Long Term Memory for Transformer Model Inference
  ☆35Jun 26, 2021Updated 4 years ago
• Open source password manager - Proton Pass • Ad
  Securely store, share, and autofill your credentials with Proton Pass, the end-to-end encrypted password manager trusted by millions.
• EleutherAI / knowledge-neurons

  View on GitHub
  A library for finding knowledge neurons in pretrained transformer models.
  ☆160Feb 13, 2022Updated 4 years ago
• seopbo / nlp_tutorials

  View on GitHub
  huggingface를 이용하여 downstream task 수행하기
  ☆62Dec 28, 2021Updated 4 years ago
• tongshoujie / MATCH-TUNING

  View on GitHub
  MATCH-TUNING
  ☆15Aug 6, 2022Updated 3 years ago
• microsoft / Efficient-Large-LM-Trainer

  View on GitHub
  ☆39Jul 25, 2024Updated last year
• salesforce / jaxformer

  View on GitHub
  Minimal library to train LLMs on TPU in JAX with pjit().
  ☆299Jun 2, 2026Updated last week
• ICLR-DAP / Deep-Audio-Prior

  View on GitHub
  Anonymous ICLR Submission
  ☆14Sep 25, 2019Updated 6 years ago
• epoch-research / training-cost-trends

  View on GitHub
  ☆25Apr 1, 2026Updated 2 months ago