leogao2/lm_dataformat external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

leogao2/lm_dataformat

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/leogao2/lm_dataformat)

Close

leogao2 / lm_dataformatView on GitHubMore

• External links
• Readme badge

☆78Dec 7, 2023Updated 2 years ago

Alternatives and similar repositories for lm_dataformat

Users that are interested in lm_dataformat are comparing it to the libraries listed below

• EleutherAI / openwebtext2

  View on GitHub
  ☆95Jul 16, 2022Updated 3 years ago
• monologg / ko_lm_dataformat

  View on GitHub
  A utility for storing and reading files for Korean LM training 💾
  ☆35Oct 15, 2025Updated 5 months ago
• EleutherAI / the-pile

  View on GitHub
  ☆1,639Apr 27, 2023Updated 2 years ago
• EleutherAI / github-downloader

  View on GitHub
  Script for downloading GitHub.
  ☆98Jul 1, 2024Updated last year
• EleutherAI / pile_dedupe

  View on GitHub
  Pile Deduplication Code
  ☆18May 15, 2023Updated 2 years ago
• EleutherAI / project-menu

  View on GitHub
  See the issue board for the current status of active and prospective projects!
  ☆65Feb 12, 2022Updated 4 years ago
• tlkh / t2t-tuner

  View on GitHub
  Convenient Text-to-Text Training for Transformers
  ☆19Dec 10, 2021Updated 4 years ago
• fastai / fastpy

  View on GitHub
  An easy way to start a python programming environment using GitHub Codespaces.
  ☆15Sep 9, 2020Updated 5 years ago
• lassl / lassl

  View on GitHub
  Easy Language Model Pretraining leveraging Huggingface's Transformers and Datasets
  ☆130Nov 12, 2022Updated 3 years ago
• AnswerDotAI / nbdev-index

  View on GitHub
  nbdev docs lookup for a few libraries and python itself
  ☆25Feb 9, 2026Updated last month
• zphang / minimal-gpt-neox-20b

  View on GitHub
  ☆131Jun 9, 2022Updated 3 years ago
• hpcaitech / ColossalAI-Pytorch-lightning

  View on GitHub
  ☆24Nov 22, 2022Updated 3 years ago
• mattf1n / basis-aware-threshold

  View on GitHub
  Code for the paper "Closing the Curious Case of Neural Text Degeneration"
  ☆12Apr 9, 2025Updated 11 months ago
• AIRC-KETI / Korean-Copora

  View on GitHub
  ☆14Dec 9, 2021Updated 4 years ago
• jason9693 / oslo-kogpt-finetunig

  View on GitHub
  kogpt를 oslo로 파인튜닝하는 예제.
  ☆23Aug 26, 2022Updated 3 years ago
• jenni-ai / T2FW

  View on GitHub
  Fine-Tuning Pre-trained Transformers into Decaying Fast Weights
  ☆19Oct 9, 2022Updated 3 years ago
• AranKomat / Metroplex

  View on GitHub
  ☆21Mar 15, 2023Updated 3 years ago
• nlpai-lab / Korean-CommonGen

  View on GitHub
  [Findings of NAACL2022] A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation
  ☆11May 27, 2022Updated 3 years ago
• EleutherAI / semantic-memorization

  View on GitHub
  ☆44Nov 17, 2024Updated last year
• EleutherAI / radioactive-lab

  View on GitHub
  Adapting the "Radioactive Data" paper to work for text models
  ☆12Dec 23, 2020Updated 5 years ago
• Beomi / transformers-language-modeling

  View on GitHub
  Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3
  ☆23May 20, 2021Updated 4 years ago
• sdtblck / youtube_subtitle_dataset

  View on GitHub
  YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training
  ☆47Sep 22, 2020Updated 5 years ago
• XinyuHua / textgen-emnlp19

  View on GitHub
  Code for our EMNLP 2019 paper titled "Sentence-Level Content Planning and Style Specification for Neural Text Generation"
  ☆17May 4, 2020Updated 5 years ago
• zphang / minimal-opt

  View on GitHub
  ☆67Aug 24, 2022Updated 3 years ago
• kyegomez / MobileVLM

  View on GitHub
  Implementation of the LDP module block in PyTorch and Zeta from the paper: "MobileVLM: A Fast, Strong and Open Vision Language Assistant …
  ☆15Mar 11, 2024Updated 2 years ago
• fastai / docments

  View on GitHub
  Document parameters using comments
  ☆10Aug 6, 2021Updated 4 years ago
• EleutherAI / tokengrams

  View on GitHub
  Efficiently computing & storing token n-grams from large corpora
  ☆27Oct 6, 2024Updated last year
• LIJUNYI95 / SuperAdam

  View on GitHub
  Official Pytorch Implementation for the paper 'SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients'
  ☆17Jan 12, 2022Updated 4 years ago
• AIRC-KETI / ke-t5-downstreams

  View on GitHub
  ☆39Mar 25, 2024Updated last year
• EleutherAI / knowledge-neurons

  View on GitHub
  A library for finding knowledge neurons in pretrained transformer models.
  ☆159Feb 13, 2022Updated 4 years ago
• HeegyuKim / korouge

  View on GitHub
  Google 공식 Rouge Implementation을 한국어에서 사용할 수 있도록 처리
  ☆18Jan 3, 2024Updated 2 years ago
• seopbo / nlp_tutorials

  View on GitHub
  huggingface를 이용하여 downstream task 수행하기
  ☆62Dec 28, 2021Updated 4 years ago
• tongshoujie / MATCH-TUNING

  View on GitHub
  MATCH-TUNING
  ☆15Aug 6, 2022Updated 3 years ago
• microsoft / Efficient-Large-LM-Trainer

  View on GitHub
  ☆39Jul 25, 2024Updated last year
• salesforce / jaxformer

  View on GitHub
  Minimal library to train LLMs on TPU in JAX with pjit().
  ☆301Dec 20, 2023Updated 2 years ago
• tylerachang / goldfish

  View on GitHub
  Goldfish: Monolingual language models for 350 languages.
  ☆24Mar 4, 2026Updated 2 weeks ago
• seujung / kolang-t5-base

  View on GitHub
  T5-base model for Korean
  ☆27May 20, 2021Updated 4 years ago
• epoch-research / training-cost-trends

  View on GitHub
  ☆24Jan 27, 2026Updated last month
• paust-team / pko-t5

  View on GitHub
  bpe based korean t5 model for text-to-text unified framework
  ☆63Apr 17, 2024Updated last year