# LLM Post-Training: A Comprehensive Reading List A structured curriculum covering everything that happens after pre-training: supervised fine-tuning, alignment, reinforcement learning from human feedback, preference optimization, parameter-efficient methods, quantization, evaluation, and safety. --- ## Phase 0: Prerequisites Before diving into post-training, you need solid grounding in transformers and how language models work at inference time. - **Vaswani et al., "Attention Is All You Need" (2017)** -- The transformer architecture paper. You need to understand self-attention, positional encoding, and encoder-decoder structure before anything else makes sense. - **Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019)** -- Establishes the decoder-only paradigm. Post-training is what turns these into useful tools. - **Andrej Karpathy, "Let's build GPT from scratch" (video, 2023)** -- Best single resource for building intuition about how autoregressive language models actually work at the code level. - **Sutton & Barto, "Reinforcement Learning: An Introduction" -- Chapters 1-3, 13** -- You need basic RL vocabulary (policy, reward, value function, policy gradient) before RLHF will make sense. - **Schulman et al., "Proximal Policy Optimization Algorithms" (2017)** -- PPO is the workhorse algorithm behind RLHF. Read this before the InstructGPT paper. --- ## Phase 1: Foundations of Post-Training ### 1.1 Supervised Fine-Tuning (SFT) SFT is the simplest form of post-training: take a pre-trained model, give it (input, output) pairs of desired behavior, train with standard cross-entropy loss. - **Howard & Ruder, "Universal Language Model Fine-tuning for Text Classification" (ULMFiT, 2018)** -- Introduced discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing. The conceptual ancestor of modern SFT practices. - **Hugging Face Transformers documentation on fine-tuning** -- Practical starting point. Walk through actually fine-tuning a model before reading more theory. ### 1.2 Instruction Tuning The insight that fine-tuning on instructions produces models that generalize to new instructions they have never seen. - **Wei et al., "Finetuned Language Models Are Zero-Shot Learners" (FLAN, 2022)** -- Fine-tune on many tasks phrased as instructions, and the model learns to follow novel instructions. Conceptual foundation for all instruction-following models. - **Chung et al., "Scaling Instruction-Finetuned Language Models" (Flan-T5/PaLM, 2022)** -- Follow-up showing that scaling both the number of tasks and model size improves instruction following. - **Sanh et al., "Multitask Prompted Training Enables Zero-Shot Task Generalization" (T0, 2022)** -- Parallel work from BigScience showing the same instruction tuning insight. - **Taori et al., "Stanford Alpaca: An Instruction-Following LLaMA Model" (2023)** -- Demonstrated that you can distill instruction-following behavior from GPT-3.5 into LLaMA 7B with only 52K examples. - **Longpre et al., "The Flan Collection: Designing Data and Methods for Effective Instruction Tuning" (2023)** -- Deep dive into what makes instruction tuning data effective. ### 1.3 The InstructGPT / RLHF Pipeline This is the core of modern alignment. Read these in order. - **Christiano et al., "Deep Reinforcement Learning from Human Preferences" (2017)** -- The original RLHF paper, applied to Atari and MuJoCo. Establishes the framework: learn a reward model from human comparisons, then optimize a policy against it. - **Stiennon et al., "Learning to Summarize from Human Feedback" (2020)** -- First application of RLHF to language models. Introduces the three-phase pipeline that InstructGPT would later scale. - **Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT, 2022)** -- The paper behind ChatGPT's training methodology. Describes the full pipeline: SFT on demonstrations, reward model from comparisons, PPO against the reward model with a KL penalty. The single most important paper on this list. - **Ziegler et al., "Fine-Tuning Language Models from Human Preferences" (2019)** -- Earlier work applying reward learning to stylistic text generation. ### 1.4 Understanding Reward Modeling - **Askell et al., "A General Language Assistant as a Laboratory for Alignment" (2021)** -- Anthropic's early work. Introduces the HHH framework and discusses reward model design choices. - **Gao et al., "Scaling Laws for Reward Model Overoptimization" (2023)** -- Critical paper. Shows that as you optimize harder against a reward model, performance on the true objective eventually degrades. You cannot do RLHF well without understanding this. --- ## Phase 2: Modern Alignment Methods ### 2.1 Direct Preference Optimization (DPO) and Variants DPO eliminates the reward model and PPO entirely, training directly on preference pairs. - **Rafailov et al., "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model" (DPO, 2023)** -- The key insight: the optimal RLHF policy has a closed-form relationship to the reward function, so you can skip reward modeling and PPO entirely. Elegant and practically important. - **Azar et al., "A General Theoretical Paradigm to Understand Learning from Human Feedback" (IPO, 2023)** -- Identifies a subtle flaw in DPO's derivation. Proposes Identity Preference Optimization as a fix. - **Ethayarajh et al., "KTO: Model Alignment as Prospect Theoretic Optimization" (2024)** -- Works with binary feedback (good/bad) instead of pairwise comparisons. - **Hong et al., "ORPO: Monolithic Preference Optimization without Reference Model" (2024)** -- Combines SFT and preference optimization into a single training phase. - **Tunstall et al., "Zephyr: Direct Distillation of LM Alignment" (2023)** -- Practical application of DPO to distill alignment from a large model into Mistral-7B. ### 2.2 Constitutional AI and RLAIF - **Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (2022)** -- Anthropic's approach. The model critiques and revises its own outputs according to a set of principles. Introduces RLAIF. - **Lee et al., "RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (2023)** -- AI-generated preference labels can match or approach human-generated labels. - **Bai et al., "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback" (2022)** -- Detailed analysis of the tension between helpfulness and harmlessness. ### 2.3 Reasoning via Reinforcement Learning - **DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025)** -- Landmark paper demonstrating that pure RL can elicit sophisticated reasoning behavior. The model develops its own reasoning patterns through GRPO. Essential reading. - **DeepSeek-AI, "DeepSeek-R1-Zero"** -- The zero-shot RL variant. Shows emergent chain-of-thought without any supervised reasoning examples. - **Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (2024)** -- Introduces GRPO (Group Relative Policy Optimization), the algorithm later used in DeepSeek-R1. --- ## Phase 3: Parameter-Efficient Methods and Practical Techniques ### 3.1 LoRA and Adapters - **Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021)** -- The dominant parameter-efficient fine-tuning method. Decomposes weight updates into low-rank matrices. You will use this constantly. - **Dettmers et al., "QLoRA: Efficient Finetuning of Quantized Language Models" (2023)** -- Combines 4-bit quantization with LoRA adapters. Makes fine-tuning 65B parameter models possible on a single 48GB GPU. - **Houlsby et al., "Parameter-Efficient Transfer Learning for NLP" (2019)** -- The original adapter paper. LoRA largely superseded this approach. - **Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation" (2024)** -- Decomposes weights into magnitude and direction, applying LoRA only to the direction component. ### 3.2 Quantization - **Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers" (2022)** -- One-shot weight quantization to 3-4 bits. The standard for GPU-based quantized inference. - **Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (2023)** -- Quantizes weights based on importance to activations. - **llama.cpp and GGUF format** -- The practical standard for CPU-based quantized inference. - **Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022)** -- Introduced mixed-precision decomposition for outlier features. ### 3.3 Knowledge Distillation - **Hinton et al., "Distilling the Knowledge in a Neural Network" (2015)** -- The foundational distillation paper. Train the student on soft labels from the teacher. - **Gu et al., "MiniLLM: Knowledge Distillation of Large Language Models" (2024)** -- Addresses the specific challenges of distilling autoregressive language models. - **The DeepSeek-R1 distillation results** -- Distillation of reasoning capability from R1 into Qwen and LLaMA base models. ### 3.4 Model Merging - **Wortsman et al., "Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Time" (2022)** -- Averaging weights of models fine-tuned with different hyperparameters. - **Yadav et al., "TIES-Merging: Resolving Interference When Merging Models" (2023)** -- Addresses sign conflicts when merging task vectors. - **Goddard et al., "Arcee's MergeKit" (2024)** -- The standard open-source tool for model merging. --- ## Phase 4: Advanced Topics ### 4.1 Safety Training and Red-Teaming - **Perez et al., "Red Teaming Language Models with Language Models" (2022)** -- Using LLMs to automatically generate adversarial prompts. - **Ganguli et al., "Red Teaming Language Models to Reduce Harms" (2022)** -- Anthropic's systematic study of manual red-teaming. - **Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models" (GCG, 2023)** -- Gradient-based adversarial suffix attacks that break alignment. - **Wei et al., "Jailbroken: How Does LLM Safety Training Fail?" (2023)** -- Taxonomizes jailbreak techniques. Two fundamental failure modes: competing objectives and mismatched generalization. ### 4.2 Evaluation of Post-Trained Models - **Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023)** -- MT-Bench and the Elo-rated Chatbot Arena. - **Li et al., "AlpacaEval: An Automatic Evaluator for Instruction-Following Models" (2023)** -- Automated evaluation using GPT-4 as judge. - **Liang et al., "Holistic Evaluation of Language Models" (HELM, 2022)** -- Stanford's comprehensive evaluation framework. - **Hendrycks et al., "Measuring Massive Multitask Language Understanding" (MMLU, 2020)** -- 57-subject multiple choice benchmark. ### 4.3 Data Curation for Post-Training - **Zhou et al., "LIMA: Less Is More for Alignment" (2023)** -- Fine-tuning on just 1,000 carefully curated examples produces a strong instruction-following model. Argues alignment is mostly about surfacing pre-trained knowledge. - **Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023)** -- Meta's detailed post-training pipeline. Sections 3-4 on SFT and RLHF data collection are among the most detailed public descriptions. - **Ivison et al., "Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2" (2023)** -- Systematic study of data mixing for instruction tuning. ### 4.4 Scaling and Iterative Post-Training - **Dubey et al., "The Llama 3 Herd of Models" (2024)** -- Meta's most detailed post-training description to date. The current best public reference for industrial-scale post-training. ### 4.5 Online/Iterative RLHF and Beyond PPO - **Gulcehre et al., "Reinforced Self-Training (ReST) for Language Modeling" (2023)** -- Alternates between sampling and training on filtered high-reward samples. - **Yuan et al., "Self-Rewarding Language Models" (2024)** -- The model acts as its own reward model, creating a self-improving loop. - **Xiong et al., "Iterative Preference Learning from Human Feedback" (2024)** -- Theoretical analysis of why online/iterative DPO outperforms offline DPO. --- ## Phase 5: Practical Tools and Implementation ### 5.1 Frameworks - **Hugging Face TRL (Transformer Reinforcement Learning)** -- The standard library for SFT, reward modeling, DPO, PPO, and KTO. - github.com/huggingface/trl - **OpenRLHF** -- Full RLHF pipeline with Ray-based distributed training. Better than TRL for large-scale PPO training. Supports GRPO. - github.com/OpenRLHF/OpenRLHF - **Axolotl** -- Config-driven fine-tuning framework. Supports LoRA/QLoRA/full fine-tuning. The easiest path from zero to a fine-tuned model. - github.com/axolotl-ai-cloud/axolotl - **LLaMA-Factory** -- Config-driven fine-tuning framework with a web UI. Supports SFT, RLHF, DPO, and various PEFT methods. - github.com/hiyouga/LLaMA-Factory ### 5.2 Datasets - **Open Assistant Conversations (OASST)** -- Human-generated conversation trees with rankings. - **Anthropic HH-RLHF** -- Human preference data for helpfulness and harmlessness. - **UltraFeedback** -- Large-scale AI feedback dataset with GPT-4 annotations. ### 5.3 Suggested Practical Projects (in order of difficulty) 1. Fine-tune a 7B model on a small instruction dataset using Axolotl with QLoRA. Evaluate on MT-Bench. 2. Train a DPO model using TRL on the Anthropic HH-RLHF dataset. Compare against the SFT-only baseline. 3. Train a reward model and run PPO using TRL or OpenRLHF. Observe reward hacking firsthand. 4. Reproduce a simplified version of the DeepSeek-R1 approach: apply GRPO to a math reasoning task using OpenRLHF. 5. Merge two LoRA fine-tunes with MergeKit and evaluate whether the merged model retains capabilities from both. --- ## Recommended Reading Order For someone starting from scratch: 1. Karpathy's GPT video (build intuition) 2. PPO paper (learn the RL algorithm) 3. Stiennon et al. 2020 (RLHF applied to summarization) 4. Ouyang et al. 2022 (InstructGPT -- the full RLHF pipeline) 5. Wei et al. 2022 (FLAN -- instruction tuning) 6. Rafailov et al. 2023 (DPO -- the modern alternative to PPO) 7. Bai et al. 2022 (Constitutional AI) 8. Hu et al. 2021 (LoRA) 9. Dettmers et al. 2023 (QLoRA) 10. Zhou et al. 2023 (LIMA -- data quality matters) 11. Touvron et al. 2023 (Llama 2 -- industrial post-training) 12. DeepSeek-R1 2025 (reasoning via RL) 13. Gao et al. 2023 (reward model overoptimization -- understanding failure modes) 14. Hands-on: Axolotl QLoRA fine-tune, then TRL DPO training --- ## Key Surveys - **Wang et al., "Aligning Large Language Models with Human: A Survey" (2023)** -- Comprehensive survey of alignment methods. - **Shen et al., "Large Language Model Alignment: A Survey" (2023)** -- Thorough coverage of the RLHF/DPO landscape. - **Sebastian Raschka, "LLM Fine-Tuning and RLHF" (blog posts and newsletter)** -- Consistently clear explanations of new post-training techniques.