Alternatives and similar repositories for Position-Prediction-Pretraining:

Users that are interested in Position-Prediction-Pretraining are comparing it to the libraries listed below

• smonsays / hypernetwork-attention
  Official code for the paper "Attention as a Hypernetwork"
  ☆25Updated 9 months ago
• formll / resolving-scaling-law-discrepancies
  ☆18Updated 8 months ago
• RobertCsordas / linear_layer_as_attention
  The official repository for our paper "The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns …
  ☆16Updated last year
• IdoAmos / not-from-scratch
  ☆31Updated 5 months ago
• lapisrocks / DiscreteAdversarialDistillation
  [NeurIPS 2023] Official repository for "Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models"
  ☆12Updated 9 months ago
• radarFudan / Curse-of-memory
  Curse-of-memory phenomenon of RNNs in sequence modelling
  ☆19Updated last week
• minhtannguyen / transformer-mgk
  This is the public github for our paper "Transformer with a Mixture of Gaussian Keys"
  ☆26Updated 2 years ago
• uclaml / MoE
  Towards Understanding the Mixture-of-Experts Layer in Deep Learning
  ☆25Updated last year
• facebookresearch / iclmlp
  Experiments for "A Closer Look at In-Context Learning under Distribution Shifts"
  ☆20Updated last year
• locuslab / T-MARS
  Code for T-MARS data filtering
  ☆35Updated last year
• lsj2408 / URPE
  [NeurIPS 2022] Your Transformer May Not be as Powerful as You Expect (official implementation)
  ☆34Updated last year
• LukasHedegaard / structured-pruning-adapters
  Structured Pruning Adapters in PyTorch
  ☆16Updated last year
• wang-kee / LiNeS
  Official repository of "LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging"
  ☆24Updated 4 months ago
• twinkle0331 / LGTM
  [ACL 2023] Code for paper “Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation”(https://arxiv.org/abs/2305.…
  ☆38Updated last year
• duykhuongnguyen / LASeR-MAB
  Code for paper: "LASeR: Learning to Adaptively Select Reward Models with Multi-Arm Bandits"
  ☆13Updated 5 months ago
• wzq016 / PINE
  Offcial Repo of Paper "Eliminating Position Bias of Language Models: A Mechanistic Approach""
  ☆12Updated 7 months ago
• kaistAI / GAP
  [ACL 2023] Gradient Ascent Post-training Enhances Language Model Generalization
  ☆29Updated 6 months ago
• abaheti95 / LoL-RL
  Advantage Leftover Lunch Reinforcement Learning (A-LoL RL): Improving Language Models with Advantage-based Offline Policy Gradients
  ☆26Updated 6 months ago
• cassidylaidlaw / orpo
  ☆14Updated 4 months ago
• OpenNLPLab / HGRN2
  HGRN2: Gated Linear RNNs with State Expansion
  ☆53Updated 7 months ago
• yale-nlp / refdpo
  ☆15Updated 8 months ago
• UKPLab / iclr2024-model-merging
  This is the repository for "Model Merging by Uncertainty-Based Gradient Matching", ICLR 2024.
  ☆27Updated 10 months ago
• janphilippfranken / sami
  Self-Supervised Alignment with Mutual Information
  ☆16Updated 10 months ago
• alexrame / diwa
  DiWA: Diverse Weight Averaging for Out-of-Distribution Generalization
  ☆29Updated 2 years ago
• alextamkin / active-learning-pretrained-models
  Active Learning Helps Pretrained Models Learn the Intended Task (https://arxiv.org/abs/2204.08491) by Alex Tamkin, Dat Nguyen, Salil Desh…
  ☆11Updated 2 years ago
• annosubmission / GRC-Cache
  ☆16Updated 2 years ago
• yifanzhang-pro / M-MAE
  Official implementation of Matrix Variational Masked Autoencoder (M-MAE) for ICML paper "Information Flow in Self-Supervised Learning" (h…
  ☆14Updated 6 months ago
• JeanKaddour / LAWA
  Latest Weight Averaging (NeurIPS HITY 2022)
  ☆29Updated last year
• facebookresearch / ViP-MAE
  This is a PyTorch implementation of the paperViP A Differentially Private Foundation Model for Computer Vision
  ☆36Updated last year
• zhjohnchan / bert-clip-synesthesia
  [Findings of ACL-2023] This is the official implementation of On the Difference of BERT-style and CLIP-style Text Encoders.
  ☆14Updated last year