Alternatives and similar repositories for Position-Prediction-Pretraining

Users that are interested in Position-Prediction-Pretraining are comparing it to the libraries listed below

• DaehanKim / EasyRLHF
  EasyRLHF aims to provide an easy and minimal interface to train aligned language models, using off-the-shelf solutions and datasets
  ☆9Updated last year
• jason9693 / FROZEN
  ☆14Updated 3 years ago
• RobertCsordas / linear_layer_as_attention
  The official repository for our paper "The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns …
  ☆16Updated last month
• abhishekpanigrahi1996 / transformer_in_transformer
  ☆45Updated last year
• smonsays / hypernetwork-attention
  Official code for the paper "Attention as a Hypernetwork"
  ☆40Updated last year
• JeanKaddour / LAWA
  Latest Weight Averaging (NeurIPS HITY 2022)
  ☆30Updated 2 years ago
• lsj2408 / URPE
  [NeurIPS 2022] Your Transformer May Not be as Powerful as You Expect (official implementation)
  ☆34Updated last year
• IdoAmos / not-from-scratch
  ☆32Updated 8 months ago
• r-three / RAD
  Reference implementation for Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
  ☆43Updated last year
• JeanKaddour / NoTrainNoGain
  Revisiting Efficient Training Algorithms For Transformer-based Language Models (NeurIPS 2023)
  ☆80Updated last year
• gregorbachmann / scaling_mlps
  ☆51Updated last year
• krafton-ai / mambaformer-icl
  MambaFormer in-context learning experiments and implementation for https://arxiv.org/abs/2402.04248
  ☆55Updated last year
• alinlab / HOMER
  Official implementation of Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs (ICLR 2024).
  ☆42Updated 11 months ago
• zhjohnchan / bert-clip-synesthesia
  [Findings of ACL-2023] This is the official implementation of On the Difference of BERT-style and CLIP-style Text Encoders.
  ☆14Updated 2 years ago
• wzq016 / PINE
  Offcial Repo of Paper "Eliminating Position Bias of Language Models: A Mechanistic Approach""
  ☆14Updated last month
• kaistAI / GAP
  [ACL 2023] Gradient Ascent Post-training Enhances Language Model Generalization
  ☆29Updated 10 months ago
• LukasHedegaard / structured-pruning-adapters
  Structured Pruning Adapters in PyTorch
  ☆18Updated last year
• ImKeTT / AdaVAE
  [Preprint] AdaVAE: Exploring Adaptive GPT-2s in VAEs for Language Modeling PyTorch Implementation
  ☆35Updated last year
• formll / resolving-scaling-law-discrepancies
  ☆20Updated last year
• Cranial-XIX / longhorn
  Official PyTorch Implementation of the Longhorn Deep State Space Model
  ☆53Updated 7 months ago
• cloneofsimo / fim-llama-deepspeed
  ☆31Updated last year
• lucidrains / tableformer-pytorch
  Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch
  ☆39Updated 3 years ago
• science-of-finetuning / sparsity-artifacts-crosscoders
  Code for the "Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning" paper.
  ☆11Updated 2 weeks ago
• OpenNLPLab / HGRN
  [NeurIPS 2023 spotlight] Official implementation of HGRN in our NeurIPS 2023 paper - Hierarchically Gated Recurrent Neural Network for Se…
  ☆66Updated last year
• sanyalsunny111 / Early_Weight_Avg
  [COLM 2024] Early Weight Averaging meets High Learning Rates for LLM Pre-training
  ☆16Updated 9 months ago
• LCM-Lab / Bridge_Gap_Diffusion
  ☆34Updated last year
• facebookresearch / iclmlp
  Experiments for "A Closer Look at In-Context Learning under Distribution Shifts"
  ☆19Updated 2 years ago
• sustcsonglin / mamba-triton
  ☆48Updated last year
• duykhuongnguyen / LASeR-MAB
  Code for paper: "LASeR: Learning to Adaptively Select Reward Models with Multi-Arm Bandits"
  ☆13Updated 9 months ago
• naver-ai / model-stock
  Model Stock: All we need is just a few fine-tuned models
  ☆118Updated 9 months ago