leogao2/lm_dataformat external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

leogao2/lm_dataformat

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/leogao2/lm_dataformat)

Close

leogao2 / lm_dataformatView on GitHubMore

• External links
• Readme badge

☆78Dec 7, 2023Updated 2 years ago

Alternatives and similar repositories for lm_dataformat

Users that are interested in lm_dataformat are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• monologg / ko_lm_dataformat

  View on GitHub
  A utility for storing and reading files for Korean LM training 💾
  ☆35Oct 15, 2025Updated 6 months ago
• EleutherAI / the-pile

  View on GitHub
  ☆1,646Apr 27, 2023Updated 3 years ago
• EleutherAI / github-downloader

  View on GitHub
  Script for downloading GitHub.
  ☆99Jul 1, 2024Updated last year
• EleutherAI / pile_dedupe

  View on GitHub
  Pile Deduplication Code
  ☆18May 15, 2023Updated 2 years ago
• EleutherAI / project-menu

  View on GitHub
  See the issue board for the current status of active and prospective projects!
  ☆65Feb 12, 2022Updated 4 years ago
• AI Agents on DigitalOcean Gradient AI Platform • Ad
  Build production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
• tlkh / t2t-tuner

  View on GitHub
  Convenient Text-to-Text Training for Transformers
  ☆18Dec 10, 2021Updated 4 years ago
• noanabeshima / wikipedia-downloader

  View on GitHub
  Downloads 2020 English Wikipedia articles as plaintext
  ☆27Mar 25, 2023Updated 3 years ago
• EleutherAI / DeeperSpeed

  View on GitHub
  DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
  ☆172Sep 26, 2025Updated 7 months ago
• fastai / fastpy

  View on GitHub
  An easy way to start a python programming environment using GitHub Codespaces.
  ☆15Sep 9, 2020Updated 5 years ago
• lassl / lassl

  View on GitHub
  Easy Language Model Pretraining leveraging Huggingface's Transformers and Datasets
  ☆130Nov 12, 2022Updated 3 years ago
• zphang / minimal-gpt-neox-20b

  View on GitHub
  ☆131Jun 9, 2022Updated 3 years ago
• hpcaitech / ColossalAI-Pytorch-lightning

  View on GitHub
  ☆24Nov 22, 2022Updated 3 years ago
• mattf1n / basis-aware-threshold

  View on GitHub
  Code for the paper "Closing the Curious Case of Neural Text Degeneration"
  ☆12Apr 9, 2025Updated last year
• EleutherAI / pyfra

  View on GitHub
  Python Research Framework
  ☆107Nov 3, 2022Updated 3 years ago
• 1-Click AI Models by DigitalOcean Gradient • Ad
  Deploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
• AIRC-KETI / Korean-Copora

  View on GitHub
  ☆14Dec 9, 2021Updated 4 years ago
• jason9693 / oslo-kogpt-finetunig

  View on GitHub
  kogpt를 oslo로 파인튜닝하는 예제.
  ☆23Aug 26, 2022Updated 3 years ago
• AranKomat / Metroplex

  View on GitHub
  ☆21Mar 15, 2023Updated 3 years ago
• nlpai-lab / Korean-CommonGen

  View on GitHub
  [Findings of NAACL2022] A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation
  ☆11May 27, 2022Updated 3 years ago
• EleutherAI / semantic-memorization

  View on GitHub
  ☆44Nov 17, 2024Updated last year
• sdtblck / youtube_subtitle_dataset

  View on GitHub
  YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training
  ☆46Sep 22, 2020Updated 5 years ago
• Beomi / transformers-language-modeling

  View on GitHub
  Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3
  ☆23May 20, 2021Updated 4 years ago
• XinyuHua / textgen-emnlp19

  View on GitHub
  Code for our EMNLP 2019 paper titled "Sentence-Level Content Planning and Style Specification for Neural Text Generation"
  ☆17May 4, 2020Updated 5 years ago
• lucidrains / all-normalization-transformer

  View on GitHub
  A simple Transformer where the softmax has been replaced with normalization
  ☆20Sep 11, 2020Updated 5 years ago
• 1-Click AI Models by DigitalOcean Gradient • Ad
  Deploy popular AI models on DigitalOcean Gradient GPU virtual machines with just a single click. Zero configuration with optimized deployments.
• zphang / minimal-opt

  View on GitHub
  ☆67Aug 24, 2022Updated 3 years ago
• kyegomez / MobileVLM

  View on GitHub
  Implementation of the LDP module block in PyTorch and Zeta from the paper: "MobileVLM: A Fast, Strong and Open Vision Language Assistant …
  ☆15Mar 11, 2024Updated 2 years ago
• fastai / docments

  View on GitHub
  Document parameters using comments
  ☆10Aug 6, 2021Updated 4 years ago
• AIRC-KETI / ke-t5-downstreams

  View on GitHub
  ☆39Mar 25, 2024Updated 2 years ago
• AeroScripts / HiddenEngrams

  View on GitHub
  Hidden Engrams: Long Term Memory for Transformer Model Inference
  ☆35Jun 26, 2021Updated 4 years ago
• EleutherAI / knowledge-neurons

  View on GitHub
  A library for finding knowledge neurons in pretrained transformer models.
  ☆159Feb 13, 2022Updated 4 years ago
• HeegyuKim / korouge

  View on GitHub
  Google 공식 Rouge Implementation을 한국어에서 사용할 수 있도록 처리
  ☆17Jan 3, 2024Updated 2 years ago
• tongshoujie / MATCH-TUNING

  View on GitHub
  MATCH-TUNING
  ☆15Aug 6, 2022Updated 3 years ago
• microsoft / Efficient-Large-LM-Trainer

  View on GitHub
  ☆39Jul 25, 2024Updated last year
• Wordpress hosting with auto-scaling - Free Trial Offer • Ad
  Fully Managed hosting for WordPress and WooCommerce businesses that need reliable, auto-scalable performance. Cloudways SafeUpdates now available.
• salesforce / jaxformer

  View on GitHub
  Minimal library to train LLMs on TPU in JAX with pjit().
  ☆299Dec 20, 2023Updated 2 years ago
• epoch-research / training-cost-trends

  View on GitHub
  ☆25Apr 1, 2026Updated last month
• seujung / kolang-t5-base

  View on GitHub
  T5-base model for Korean
  ☆27May 20, 2021Updated 4 years ago
• JunShern / few-shot-adaptation

  View on GitHub
  Exploring Few-Shot Adaptation of Language Models with Tables
  ☆24Aug 22, 2022Updated 3 years ago
• paust-team / pko-t5

  View on GitHub
  bpe based korean t5 model for text-to-text unified framework
  ☆63Apr 17, 2024Updated 2 years ago
• lucidrains / token-shift-gpt

  View on GitHub
  Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing
  ☆49Jan 27, 2022Updated 4 years ago
• huggingface / datablations

  View on GitHub
  Scaling Data-Constrained Language Models
  ☆343Jun 28, 2025Updated 10 months ago