A structured curriculum for going from zero to being able to read and engage with current alignment research. Each phase builds on the last. Do not skip phases unless you already have the background.
Estimated total time: 6-12 months of serious part-time study.
Before diving into technical material, get the lay of the land. These give you the vocabulary and motivation to understand why alignment matters and what the field is trying to do.
Robert Miles YouTube Channel (youtube.com/@RobertMilesAI) -- Start here. Watch the "AI Safety" playlist in order. Miles is the best communicator in the field for newcomers. His videos on mesa-optimization, instrumental convergence, and reward hacking will give you intuitions that textbooks take chapters to build.
80,000 Hours: "Preventing an AI-related catastrophe" (80000hours.org/problem-profiles/artificial-intelligence/) -- A sober, well-sourced overview of why AI safety is considered a pressing problem. Read the full profile, not just the summary.
Wait But Why: "The AI Revolution" by Tim Urban (waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html) -- Parts 1 and 2. Dated in some specifics but still the most accessible introduction to the core concern. Gets the exponential growth intuition across better than anything else.
"AGI Safety Fundamentals" Curriculum (agisafetyfundamentals.com/) -- The AI Safety Fundamentals (formerly AGI Safety Fundamentals, or AGISF) course provides a structured 8-week reading list with facilitated discussion groups. You can use the syllabus as a self-study guide even without joining a cohort. The "Alignment" track is the relevant one.
The Sequences are the intellectual foundation of the rationalist-adjacent alignment community. They are long. They are worth it. Reading them teaches you how alignment researchers think, which is as important as the object-level content.
Available at readthesequences.com or on LessWrong (lesswrong.com/rationality). Read them in the order below.
Predictably Wrong -- Cognitive biases are not random noise; they are systematic. This sequence introduces the idea that human reasoning has reliable failure modes, which matters because alignment is fundamentally about specifying what we want, and we are bad at knowing what we want.
Fake Beliefs -- The distinction between beliefs you actually act on and beliefs you merely profess. Critical for understanding why "just tell the AI to be good" fails.
Noticing Confusion -- How to notice when your model of the world is wrong. Trains the mental habit of treating surprise as information.
Mysterious Answers -- Why "emergence" and "complexity" are not explanations. Teaches you to reject fake explanations, which is essential for evaluating alignment proposals.
Overly Convenient Excuses -- How motivated reasoning protects bad ideas from evidence.
Politics is the Mind-Killer -- Why political thinking destroys epistemic rationality. Relevant because AI policy debates are politically charged.
Against Rationalization -- The difference between arriving at a conclusion through reasoning versus constructing reasoning to support a conclusion you already hold.
Against Doublethink -- On the cost of maintaining contradictory beliefs.
Seeing with Fresh Eyes -- How to actually update on evidence instead of anchoring.
Death Spirals -- How groups and individuals get trapped in escalating commitment to wrong ideas. Relevant to evaluating the alignment field itself.
Letting Go -- The emotional mechanics of changing your mind. Sounds soft, but this is the hardest part of rationality in practice.
The Simple Math of Evolution -- Evolution as an optimization process. This is the first bridge to AI: understanding optimization processes that are powerful but have no values.
Fragility of Value -- Why "close enough" values in an optimizer are not close enough. One of the most important sequences for alignment specifically. The thesis: human values are a tiny target in the space of possible objectives, and small deviations lead to catastrophe.
A Human's Guide to Words -- Philosophy of language and categorization. Teaches you to notice when alignment debates are actually about definitions rather than substance.
Lawful Truth -- The universe runs on math, not narratives. Foundation for taking a technical approach to AI safety.
Reductionism 101 -- Why higher-level descriptions reduce to lower-level ones, and why this matters for understanding minds and values.
Joy in the Merely Real -- Short and motivational. You can skim this.
Physicalism 201 -- Deeper on reductionism. The zombie thought experiment and why consciousness debates matter less than you think for alignment.
Quantum Physics -- You can skip most of this. The payoff is understanding many-worlds, but this is not load-bearing for alignment.
Fake Preferences -- Why stated preferences diverge from revealed preferences, and what this means for value alignment.
Value Theory -- What values are, how they interact, and why they resist simple formalization.
Quantified Humanism -- Consequentialism, scope insensitivity, and why feelings are unreliable guides to ethics at scale.
The Craft and the Community -- How to build and participate in intellectual communities. You can skim this unless you plan to work in the field.
Yudkowsky's Coming of Age -- Autobiographical. Optional but gives useful context on how the field started.
Read these in parallel with or after the Sequences. They cover the same territory from different angles and with different strengths.
Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014) -- The canonical statement of the alignment problem at a strategic level. Covers takeoff scenarios, the control problem, and why alignment is hard in principle. The middle chapters on convergent instrumental goals and the treacherous turn are essential. Read chapters 1-10 carefully; skim 11-15.
Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (2019) -- Russell's alternative framing: the problem is not that AI might go rogue, but that we do not know how to specify objectives. His "three principles" for beneficial AI (uncertainty about objectives, deference to humans, learning preferences from behavior) are influential. More accessible than Bostrom and more technically grounded.
Brian Christian, The Alignment Problem: Machine Learning and Human Values (2020) -- The best-written of the three. Covers the history of attempts to align ML systems with human intentions, from fairness in criminal justice algorithms to reward hacking in RL. Less focused on existential risk, more on the concrete ways alignment fails today. Good for building intuition about why the problem is hard even in mundane settings.
You need enough ML background to read papers. If you already have this, skip ahead.
3Blue1Brown: "Neural Networks" series -- Visual intuition for how neural networks work. Four videos. Watch first.
Andrej Karpathy: "Let's build GPT" -- Build a transformer from scratch. This single video gives you more useful intuition about LLMs than most courses.
Spinning Up in Deep RL (spinningup.openai.com/) -- OpenAI's introduction to reinforcement learning. You need RL basics to understand RLHF and reward modeling. Read Part 1 (key concepts) and Part 2 (kinds of RL algorithms) at minimum.
"Risks from Learned Optimization in Advanced Machine Learning Systems" (Hubinger et al., 2019) -- The mesa-optimization paper. Introduces the distinction between the base optimizer (training) and the mesa-optimizer (the learned model). The concept of deceptive alignment comes from here. One of the most important papers in the field.
"Concrete Problems in AI Safety" (Amodei et al., 2016) -- The paper that helped legitimize AI safety as a research agenda within mainstream ML. Categorizes safety problems: safe exploration, distributional shift, reward hacking, safe interruptibility, side effects. Still a useful taxonomy.
"The Alignment Problem from a Deep Learning Perspective" (Ngo, Chan, and Lovering, 2023) -- A more recent overview that frames alignment problems in terms of current deep learning systems rather than hypothetical future AGI. Good bridge between the philosophical and technical literature.
Paul Christiano's Blog (ai-alignment.com/) -- Christiano is one of the most important alignment researchers. His blog posts on iterated amplification, eliciting latent knowledge (ELK), and market-based approaches to alignment are essential reading. Start with:
Paul Christiano, "Iterated Amplification" -- The core idea: if you cannot train an AI to be as capable as you need, you can train a sequence of AI systems where each one is slightly more capable than the last, with human oversight at each step. This is the intellectual ancestor of RLHF and constitutional AI. Read the original blog posts and the paper "Supervising strong learners by amplifying weak experts."
This is where you start reading the actual research that alignment researchers are producing and debating today.
"Training language models to follow instructions with human feedback" (Ouyang et al., 2022) -- The InstructGPT paper. This is the practical implementation of RLHF that led to ChatGPT. Understand the three-step process: supervised fine-tuning, reward model training, PPO optimization against the reward model.
"Fine-Tuning Language Models from Human Preferences" (Ziegler et al., 2019) -- Earlier RLHF work that established the paradigm.
"A General Language Assistant as a Laboratory for Alignment" (Askell et al., 2021, Anthropic) -- Anthropic's early work on training helpful, harmless, and honest assistants. Introduces the HHH framework.
"Direct Preference Optimization: Your Language Model Is Secretly a Reward Model" (Rafailov et al., 2023) -- DPO eliminates the separate reward model step in RLHF. Important because it simplifies the training pipeline and changes the failure modes.
"Constitutional AI: Harmlessness from AI Feedback" (Bai et al., 2022, Anthropic) -- The CAI paper. Instead of relying entirely on human feedback, have the AI critique and revise its own outputs according to a set of principles (the "constitution"). This is a concrete approach to scalable oversight.
"Measuring Faithfulness in Chain-of-Thought Reasoning" (Lanham et al., 2023, Anthropic) -- Investigates whether chain-of-thought reasoning actually reflects the model's decision process or is post-hoc rationalization. This matters because if CoT is unfaithful, we cannot use it for oversight.
"Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training" (Hubinger et al., 2024, Anthropic) -- Demonstrates that current safety training techniques (RLHF, adversarial training) fail to remove backdoor behaviors from models trained to be deceptive. A concrete empirical result on the difficulty of alignment.
"Weak-to-Strong Generalization" (Burns et al., 2023, OpenAI) -- Studies whether weak models can supervise stronger models, which is a simplified version of the core alignment challenge (humans supervising superhuman AI).
Interpretability is the subfield of alignment concerned with understanding what models are actually doing internally, rather than just observing their behavior.
"Toy Models of Superposition" (Elhage et al., 2022, Anthropic) -- Foundational paper for mechanistic interpretability. Shows that neural networks represent more features than they have dimensions by encoding them in superposition. This is why interpreting neural networks is hard.
"Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" (Bricken et al., 2023, Anthropic) -- Uses sparse autoencoders to decompose model activations into interpretable features. A major step toward actually understanding what is going on inside large models.
"Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" (Templeton et al., 2024, Anthropic) -- Extends the dictionary learning approach to production-scale models. Finds features corresponding to specific concepts (Golden Gate Bridge, deception, code errors). Demonstrates that interpretability can scale.
"A Mathematical Framework for Transformer Circuits" (Elhage et al., 2021, Anthropic) -- Technical foundation for the circuits-based approach to interpretability. Read after the papers above.
Anthropic's Transformer Circuits Thread (transformer-circuits.pub/) -- The full collection of Anthropic's interpretability research. After reading the papers above, browse the rest of the thread for additional results.
Chris Olah's Blog (colah.github.io/) -- Olah leads Anthropic's interpretability work. His older blog posts on neural network visualization are classics and build intuition for the newer work.
The Machine Intelligence Research Institute was the first organization focused on alignment. Their technical agenda is more theoretical and pessimistic than most.
"MIRI's 2023 Technical Research Agenda" -- Read the latest version on their website (intelligence.org/research/). Their current focus is on understanding the nature of agency and optimization in a way that would let us predict and control advanced AI systems.
Nate Soares and Benya Fallenstein, "Agent Foundations for Aligning Machine Intelligence with Human Interests" (2017) -- Overview of MIRI's earlier research directions: logical uncertainty, Vingean reflection, naturalized induction, decision theory. Dense but important for understanding one school of thought in alignment.
Eliezer Yudkowsky, "AGI Ruin: A List of Lethalities" (2022, LessWrong) -- Yudkowsky's comprehensive argument for why alignment is extremely hard and current approaches are insufficient. Whether or not you agree with the conclusions, you need to understand this argument because it is the strongest version of the pessimistic case.
ARC was founded by Paul Christiano to work on alignment research with a focus on tractable theoretical questions.
"Eliciting Latent Knowledge" (ELK) -- ARC's central research problem. If a model has learned an accurate world model but has been trained to say things humans want to hear, how do you extract the truth? The ELK report is the most important document ARC has produced.
ARC Evals -- ARC pivoted partially toward evaluating frontier models for dangerous capabilities (self-replication, resource acquisition, deception). Their evaluation methodology is influential. Follow their publications at evals.alignment.org.
"Scalable agent alignment via reward modeling" (Leike et al., 2018) -- DeepMind's alignment research agenda, focusing on recursive reward modeling. Complements Christiano's iterated amplification.
"Goal Misgeneralization in Deep Reinforcement Learning" (Langosco et al., 2022) -- Empirical demonstration that RL agents learn goals that differ from what was intended, even when training performance is perfect. A concrete example of the mesa-optimization concern.
"Model evaluation for extreme risks" (Shevlane et al., 2023) -- Framework for evaluating whether frontier AI models pose catastrophic risks. Influential on how labs think about pre-deployment safety evaluations.
At this point you should be able to read new alignment papers as they come out and understand the arguments being made.
Alignment Forum (alignmentforum.org/) -- The primary venue for alignment research discussion. Higher signal-to-noise than LessWrong for alignment specifically.
LessWrong (lesswrong.com/) -- Broader rationalist community, but major alignment posts appear here. Follow the "AI" tag.
Anthropic Research Blog (anthropic.com/research) -- Anthropic publishes more alignment-relevant research than any other lab. Check regularly.
arXiv cs.AI and cs.LG -- Use Semantic Scholar or arXiv feeds filtered for safety/alignment keywords.
If you want to go from well-read to actively contributing:
MATS (ML Alignment Theory Scholars) (matsprogram.org/) -- A research mentorship program that pairs aspiring alignment researchers with established mentors. Competitive admission. The single best on-ramp to doing alignment research if you can get in.
ARENA (Alignment Research Engineer Accelerator) (arena.education/) -- A more engineering-focused program. Good if your background is more in software engineering than ML research.
These are people whose new work you should read as it appears:
To be literate in the current discourse, you need to understand the key disagreements, not just the consensus:
Doom vs. optimism -- Yudkowsky and MIRI argue alignment is nearly unsolvable on current trajectories. Others (Christiano, some Anthropic researchers) think it is very hard but tractable. Understanding both sides and why they disagree is essential.
Control vs. alignment -- The "AI control" agenda (Greenblatt, Shlegeris) argues we should focus on making AI systems safe to deploy even if we cannot fully align them, using monitoring, sandboxing, and redundancy. This is in tension with approaches that aim for deep alignment.
Interpretability vs. behavioral approaches -- Is it more important to understand what models are doing internally (interpretability) or to ensure they behave well across distributions (behavioral testing, RLHF, constitutional AI)? Most researchers think both matter, but they disagree on resource allocation.
Timelines -- Disagreements about when transformative AI will arrive drive disagreements about which research is worth doing. Short-timelines researchers focus on making current systems safer. Long-timelines researchers invest in more foundational theory.
If you want the fastest path to research literacy:
This condensed path takes roughly 3-4 months and gets you to the point where you can follow current research discussions.