leogao2/lm_dataformat external links

---

GitHub - View repository

GitHub.dev - Open in online code editor

Star history - Growth chart over time

deps.dev - Dependencies and security vulnerabilities

Gitingest - Export source code for LLM usage

Repomix - Export source code for LLM usage

uithub - Export source code for LLM usage

Close

leogao2/lm_dataformat

Readme badge preview -

If you own this repo, copy the snippet below and add it to your README.md

---

Copy

[![RelatedRepos](https://img.shields.io/badge/related-repos-yellow)](https://relatedrepos.com/gh/leogao2/lm_dataformat)

Close

leogao2 / lm_dataformatView on GitHubMore

• External links
• Readme badge

☆78Dec 7, 2023Updated 2 years ago

Alternatives and similar repositories for lm_dataformat

Users that are interested in lm_dataformat are comparing it to the libraries listed below. We may earn a commission when you buy through links labeled 'Ad' on this page.

• monologg / ko_lm_dataformat

  View on GitHub
  A utility for storing and reading files for Korean LM training 💾
  ☆35Jul 18, 2026Updated 3 months ago
• tlkh / t2t-tuner

  View on GitHub
  Convenient Text-to-Text Training for Transformers
  ☆18Dec 10, 2021Updated 4 years ago
• ModuNLP / hacking_transformers

  View on GitHub
  ☆11Aug 12, 2020Updated 6 years ago
• EleutherAI / the-pile

  View on GitHub
  ☆1,680Apr 27, 2023Updated 3 years ago
• EleutherAI / openwebtext2

  View on GitHub
  ☆94Jul 16, 2022Updated 4 years ago
• Deploy on Railway without the complexity - Free Credits Offer • Ad
  Connect your repo and Railway handles the rest with instant previews. Quickly provision container image services, databases, and storage volumes.
• lassl / lassl

  View on GitHub
  Easy Language Model Pretraining leveraging Huggingface's Transformers and Datasets
  ☆131Nov 12, 2022Updated 3 years ago
• EleutherAI / project-menu

  View on GitHub
  See the issue board for the current status of active and prospective projects!
  ☆65Feb 12, 2022Updated 4 years ago
• ModuNLP / weekly-meeting

  View on GitHub
  매주 목요일, 20:00 모임
  ☆16Jul 24, 2020Updated 6 years ago
• EleutherAI / github-downloader

  View on GitHub
  Script for downloading GitHub.
  ☆99Jul 1, 2024Updated 2 years ago
• EleutherAI / pile_dedupe

  View on GitHub
  Pile Deduplication Code
  ☆18May 15, 2023Updated 3 years ago
• zphang / minimal-gpt-neox-20b

  View on GitHub
  ☆131Jun 9, 2022Updated 4 years ago
• EleutherAI / pyfra

  View on GitHub
  Python Research Framework
  ☆107Nov 3, 2022Updated 3 years ago
• lucidrains / all-normalization-transformer

  View on GitHub
  A simple Transformer where the softmax has been replaced with normalization
  ☆20Sep 11, 2020Updated 6 years ago
• hpcaitech / ColossalAI-Pytorch-lightning

  View on GitHub
  ☆24Nov 22, 2022Updated 3 years ago
• Managed Kubernetes at scale on DigitalOcean • Ad
  DigitalOcean Kubernetes includes the control plane, bandwidth allowance, container registry, automatic updates, and more for free.
• Beomi / transformers-language-modeling

  View on GitHub
  Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3
  ☆23May 20, 2021Updated 5 years ago
• fastai / fastpy

  View on GitHub
  An easy way to start a python programming environment using GitHub Codespaces.
  ☆15Sep 9, 2020Updated 6 years ago
• noanabeshima / wikipedia-downloader

  View on GitHub
  Downloads 2020 English Wikipedia articles as plaintext
  ☆27Mar 25, 2023Updated 3 years ago
• sdtblck / youtube_subtitle_dataset

  View on GitHub
  YT_subtitles - extracts subtitles from YouTube videos to raw text for Language Model training
  ☆47Sep 22, 2020Updated 6 years ago
• simonjisu / annotated-transformer-kr

  View on GitHub
  annotated-transformer-kr
  ☆15May 16, 2019Updated 7 years ago
• seopbo / py-automate

  View on GitHub
  업무자동화를 위한 Python 강의를 듣고 정리한 자료
  ☆13Oct 10, 2017Updated 9 years ago
• ICLR-DAP / Deep-Audio-Prior

  View on GitHub
  Anonymous ICLR Submission
  ☆14Sep 25, 2019Updated 7 years ago
• EleutherAI / DeeperSpeed

  View on GitHub
  DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.
  ☆172Aug 23, 2026Updated 2 months ago
• EleutherAI / semantic-memorization

  View on GitHub
  ☆45Nov 17, 2024Updated last year
• AI Agents on DigitalOcean Gradient AI Platform • Ad
  Build production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
• jason9693 / ETA4LLMs

  View on GitHub
  Calculating Expected Time for training LLM.
  ☆39Apr 17, 2023Updated 3 years ago
• seopbo / nlp_tutorials

  View on GitHub
  huggingface를 이용하여 downstream task 수행하기
  ☆62Dec 28, 2021Updated 4 years ago
• AnswerDotAI / nbdev-index

  View on GitHub
  nbdev docs lookup for a few libraries and python itself
  ☆25Feb 9, 2026Updated 8 months ago
• AIRC-KETI / ke-t5-downstreams

  View on GitHub
  ☆39Mar 25, 2024Updated 2 years ago
• mattf1n / basis-aware-threshold

  View on GitHub
  Code for the paper "Closing the Curious Case of Neural Text Degeneration"
  ☆12Apr 9, 2025Updated last year
• AranKomat / Metroplex

  View on GitHub
  ☆21Mar 15, 2023Updated 3 years ago
• jason9693 / oslo-kogpt-finetunig

  View on GitHub
  kogpt를 oslo로 파인튜닝하는 예제.
  ☆23Aug 26, 2022Updated 4 years ago
• ernie-research / gptfluence

  View on GitHub
  [EMNLP'24 oral] On Training Data Influence of GPT Models
  ☆11Jun 6, 2025Updated last year
• nlpai-lab / Korean-CommonGen

  View on GitHub
  [Findings of NAACL2022] A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation
  ☆11May 27, 2022Updated 4 years ago
• AI Agents on DigitalOcean Gradient AI Platform • Ad
  Build production-ready AI agents using customizable tools or access multiple LLMs through a single endpoint. Create custom knowledge bases or connect external data.
• google-research / meliad

  View on GitHub
  ☆261Jun 6, 2025Updated last year
• kakao / kanana-2

  View on GitHub
  ☆23Jun 30, 2026Updated 3 months ago
• human-rights-corpus / HRC

  View on GitHub
  #인권코퍼스
  ☆31Oct 6, 2023Updated 3 years ago
• bitextor / warc2text

  View on GitHub
  Extracts plain text, language identification and more metadata from WARC records
  ☆22Apr 16, 2026Updated 6 months ago
• tunib-ai / oslo

  View on GitHub
  OSLO: Open Source framework for Large-scale model Optimization
  ☆308Aug 25, 2022Updated 4 years ago
• AIRC-KETI / Korean-Copora

  View on GitHub
  ☆14Dec 9, 2021Updated 4 years ago
• kyegomez / MobileVLM

  View on GitHub
  Implementation of the LDP module block in PyTorch and Zeta from the paper: "MobileVLM: A Fast, Strong and Open Vision Language Assistant …
  ☆15Mar 11, 2024Updated 2 years ago