A structured curriculum covering everything that happens after pre-training: supervised fine-tuning, alignment, reinforcement learning from human feedback, preference optimization, parameter-efficient methods, quantization, evaluation, and safety.
Before diving into post-training, you need solid grounding in transformers and how language models work at inference time.
Vaswani et al., "Attention Is All You Need" (2017) -- The transformer architecture paper. You need to understand self-attention, positional encoding, and encoder-decoder structure before anything else makes sense.
Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019) -- Establishes the decoder-only paradigm. Post-training is what turns these into useful tools.
Andrej Karpathy, "Let's build GPT from scratch" (video, 2023) -- Best single resource for building intuition about how autoregressive language models actually work at the code level.
Sutton & Barto, "Reinforcement Learning: An Introduction" -- Chapters 1-3, 13 -- You need basic RL vocabulary (policy, reward, value function, policy gradient) before RLHF will make sense.
Schulman et al., "Proximal Policy Optimization Algorithms" (2017) -- PPO is the workhorse algorithm behind RLHF. Read this before the InstructGPT paper.
SFT is the simplest form of post-training: take a pre-trained model, give it (input, output) pairs of desired behavior, train with standard cross-entropy loss.
Howard & Ruder, "Universal Language Model Fine-tuning for Text Classification" (ULMFiT, 2018) -- Introduced discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing. The conceptual ancestor of modern SFT practices.
Hugging Face Transformers documentation on fine-tuning -- Practical starting point. Walk through actually fine-tuning a model before reading more theory.
The insight that fine-tuning on instructions produces models that generalize to new instructions they have never seen.
Wei et al., "Finetuned Language Models Are Zero-Shot Learners" (FLAN, 2022) -- Fine-tune on many tasks phrased as instructions, and the model learns to follow novel instructions. Conceptual foundation for all instruction-following models.
Chung et al., "Scaling Instruction-Finetuned Language Models" (Flan-T5/PaLM, 2022) -- Follow-up showing that scaling both the number of tasks and model size improves instruction following.
Sanh et al., "Multitask Prompted Training Enables Zero-Shot Task Generalization" (T0, 2022) -- Parallel work from BigScience showing the same instruction tuning insight.
Taori et al., "Stanford Alpaca: An Instruction-Following LLaMA Model" (2023) -- Demonstrated that you can distill instruction-following behavior from GPT-3.5 into LLaMA 7B with only 52K examples.
Longpre et al., "The Flan Collection: Designing Data and Methods for Effective Instruction Tuning" (2023) -- Deep dive into what makes instruction tuning data effective.
This is the core of modern alignment. Read these in order.
Christiano et al., "Deep Reinforcement Learning from Human Preferences" (2017) -- The original RLHF paper, applied to Atari and MuJoCo. Establishes the framework: learn a reward model from human comparisons, then optimize a policy against it.
Stiennon et al., "Learning to Summarize from Human Feedback" (2020) -- First application of RLHF to language models. Introduces the three-phase pipeline that InstructGPT would later scale.
Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT, 2022) -- The paper behind ChatGPT's training methodology. Describes the full pipeline: SFT on demonstrations, reward model from comparisons, PPO against the reward model with a KL penalty. The single most important paper on this list.
Ziegler et al., "Fine-Tuning Language Models from Human Preferences" (2019) -- Earlier work applying reward learning to stylistic text generation.
Askell et al., "A General Language Assistant as a Laboratory for Alignment" (2021) -- Anthropic's early work. Introduces the HHH framework and discusses reward model design choices.
Gao et al., "Scaling Laws for Reward Model Overoptimization" (2023) -- Critical paper. Shows that as you optimize harder against a reward model, performance on the true objective eventually degrades. You cannot do RLHF well without understanding this.
DPO eliminates the reward model and PPO entirely, training directly on preference pairs.
Rafailov et al., "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model" (DPO, 2023) -- The key insight: the optimal RLHF policy has a closed-form relationship to the reward function, so you can skip reward modeling and PPO entirely. Elegant and practically important.
Azar et al., "A General Theoretical Paradigm to Understand Learning from Human Feedback" (IPO, 2023) -- Identifies a subtle flaw in DPO's derivation. Proposes Identity Preference Optimization as a fix.
Ethayarajh et al., "KTO: Model Alignment as Prospect Theoretic Optimization" (2024) -- Works with binary feedback (good/bad) instead of pairwise comparisons.
Hong et al., "ORPO: Monolithic Preference Optimization without Reference Model" (2024) -- Combines SFT and preference optimization into a single training phase.
Tunstall et al., "Zephyr: Direct Distillation of LM Alignment" (2023) -- Practical application of DPO to distill alignment from a large model into Mistral-7B.
Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (2022) -- Anthropic's approach. The model critiques and revises its own outputs according to a set of principles. Introduces RLAIF.
Lee et al., "RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (2023) -- AI-generated preference labels can match or approach human-generated labels.
Bai et al., "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback" (2022) -- Detailed analysis of the tension between helpfulness and harmlessness.
DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025) -- Landmark paper demonstrating that pure RL can elicit sophisticated reasoning behavior. The model develops its own reasoning patterns through GRPO. Essential reading.
DeepSeek-AI, "DeepSeek-R1-Zero" -- The zero-shot RL variant. Shows emergent chain-of-thought without any supervised reasoning examples.
Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (2024) -- Introduces GRPO (Group Relative Policy Optimization), the algorithm later used in DeepSeek-R1.
Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021) -- The dominant parameter-efficient fine-tuning method. Decomposes weight updates into low-rank matrices. You will use this constantly.
Dettmers et al., "QLoRA: Efficient Finetuning of Quantized Language Models" (2023) -- Combines 4-bit quantization with LoRA adapters. Makes fine-tuning 65B parameter models possible on a single 48GB GPU.
Houlsby et al., "Parameter-Efficient Transfer Learning for NLP" (2019) -- The original adapter paper. LoRA largely superseded this approach.
Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation" (2024) -- Decomposes weights into magnitude and direction, applying LoRA only to the direction component.
Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers" (2022) -- One-shot weight quantization to 3-4 bits. The standard for GPU-based quantized inference.
Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (2023) -- Quantizes weights based on importance to activations.
llama.cpp and GGUF format -- The practical standard for CPU-based quantized inference.
Dettmers et al., "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022) -- Introduced mixed-precision decomposition for outlier features.
Hinton et al., "Distilling the Knowledge in a Neural Network" (2015) -- The foundational distillation paper. Train the student on soft labels from the teacher.
Gu et al., "MiniLLM: Knowledge Distillation of Large Language Models" (2024) -- Addresses the specific challenges of distilling autoregressive language models.
The DeepSeek-R1 distillation results -- Distillation of reasoning capability from R1 into Qwen and LLaMA base models.
Wortsman et al., "Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Time" (2022) -- Averaging weights of models fine-tuned with different hyperparameters.
Yadav et al., "TIES-Merging: Resolving Interference When Merging Models" (2023) -- Addresses sign conflicts when merging task vectors.
Goddard et al., "Arcee's MergeKit" (2024) -- The standard open-source tool for model merging.
Perez et al., "Red Teaming Language Models with Language Models" (2022) -- Using LLMs to automatically generate adversarial prompts.
Ganguli et al., "Red Teaming Language Models to Reduce Harms" (2022) -- Anthropic's systematic study of manual red-teaming.
Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models" (GCG, 2023) -- Gradient-based adversarial suffix attacks that break alignment.
Wei et al., "Jailbroken: How Does LLM Safety Training Fail?" (2023) -- Taxonomizes jailbreak techniques. Two fundamental failure modes: competing objectives and mismatched generalization.
Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023) -- MT-Bench and the Elo-rated Chatbot Arena.
Li et al., "AlpacaEval: An Automatic Evaluator for Instruction-Following Models" (2023) -- Automated evaluation using GPT-4 as judge.
Liang et al., "Holistic Evaluation of Language Models" (HELM, 2022) -- Stanford's comprehensive evaluation framework.
Hendrycks et al., "Measuring Massive Multitask Language Understanding" (MMLU, 2020) -- 57-subject multiple choice benchmark.
Zhou et al., "LIMA: Less Is More for Alignment" (2023) -- Fine-tuning on just 1,000 carefully curated examples produces a strong instruction-following model. Argues alignment is mostly about surfacing pre-trained knowledge.
Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023) -- Meta's detailed post-training pipeline. Sections 3-4 on SFT and RLHF data collection are among the most detailed public descriptions.
Ivison et al., "Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2" (2023) -- Systematic study of data mixing for instruction tuning.
Gulcehre et al., "Reinforced Self-Training (ReST) for Language Modeling" (2023) -- Alternates between sampling and training on filtered high-reward samples.
Yuan et al., "Self-Rewarding Language Models" (2024) -- The model acts as its own reward model, creating a self-improving loop.
Xiong et al., "Iterative Preference Learning from Human Feedback" (2024) -- Theoretical analysis of why online/iterative DPO outperforms offline DPO.
Hugging Face TRL (Transformer Reinforcement Learning) -- The standard library for SFT, reward modeling, DPO, PPO, and KTO.
OpenRLHF -- Full RLHF pipeline with Ray-based distributed training. Better than TRL for large-scale PPO training. Supports GRPO.
Axolotl -- Config-driven fine-tuning framework. Supports LoRA/QLoRA/full fine-tuning. The easiest path from zero to a fine-tuned model.
LLaMA-Factory -- Config-driven fine-tuning framework with a web UI. Supports SFT, RLHF, DPO, and various PEFT methods.
For someone starting from scratch: