From first principles to current research. This list covers architecture, capabilities, and the broader LLM landscape. Pre-training and post-training each have their own dedicated lists.
You need just enough ML/DL fundamentals to not be lost when you open "Attention Is All You Need." Do not spend months here. Get the intuition, move on, and revisit as needed.
3Blue1Brown, "Neural Networks" series (YouTube, 4 videos) -- The best visual introduction to what neural networks actually compute. Covers neurons, layers, backpropagation, and gradient descent with genuine geometric intuition. Watch all four before touching anything else.
Andrej Karpathy, "The spelled-out intro to neural networks and backpropagation: building micrograd" (YouTube) -- Builds an autograd engine from scratch in Python. After 3Blue1Brown gives you the pictures, Karpathy gives you the code. You will understand backpropagation mechanistically, not just conceptually.
Andrej Karpathy, "The spelled-out intro to language modeling: building makemore" (YouTube, 5-part series) -- Builds character-level language models from bigrams through MLPs through RNNs. This is the bridge from "I understand neural networks" to "I understand language modeling." Covers embeddings, softmax, cross-entropy loss, and sampling.
Jay Alammar, "The Illustrated Transformer" (blog post) -- The single best companion piece to the original paper. Walks through the architecture with diagrams that make the data flow legible. Read this side-by-side with the paper.
Jay Alammar, "The Illustrated GPT-2" (blog post) -- Extends the transformer explanation to decoder-only autoregressive models. Important because most modern LLMs are decoder-only, not encoder-decoder.
Jay Alammar, "Visualizing A Neural Machine Translation Model" (blog post) -- Covers the attention mechanism with animations. Useful if the attention diagrams in the original paper feel static.
Lilian Weng, "The Transformer Family" (blog post, updated versions) -- A comprehensive catalog of transformer variants and modifications. Good as a map of the territory once you understand the base architecture.
Andrej Karpathy, "Let's build GPT: from scratch, in code, spelled out" (YouTube) -- Builds a GPT from an empty file. The single most important resource on this list. After watching this, you will understand tokenization, attention, layer norms, residual connections, and text generation at the level of individual tensor operations. Do not skip this.
Andrej Karpathy, "Let's build the GPT Tokenizer" (YouTube) -- Companion to the above. Tokenization determines what the model can and cannot represent. Byte-pair encoding, the vocabulary, special tokens.
Read these in roughly chronological order. Each paper represents a distinct design choice that shaped the field.
Radford et al., "Improving Language Understanding by Generative Pre-Training" (2018) -- GPT-1 -- Established that unsupervised pre-training on text followed by supervised fine-tuning works remarkably well. Short paper, historically important.
Radford et al., "Language Models are Unsupervised Multitask Learners" (2019) -- GPT-2 -- The key insight: scale up the model and data, and the model learns to perform tasks without explicit fine-tuning. Introduced zero-shot task performance.
Brown et al., "Language Models are Few-Shot Learners" (2020) -- GPT-3 -- 175B parameters. Demonstrated in-context learning: the model can perform new tasks given only a few examples in the prompt, with no gradient updates. Read carefully, especially the few-shot evaluation methodology.
OpenAI, "GPT-4 Technical Report" (2023) -- Light on architecture details but important for understanding capability evaluations, safety testing, and how frontier models are assessed.
Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (2023) -- Showed that smaller models trained on more data can match or beat larger models trained on less data. Launched the open-source LLM ecosystem.
Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023) -- Covers both pre-training and the RLHF pipeline. Important for understanding the full stack from base model to chat model.
Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017) -- The original MoE paper for neural networks. Conditional computation: not all parameters are active for every input.
Jiang et al., "Mixtral of Experts" (2024) -- Applies sparse MoE to the transformer architecture cleanly. MoE is increasingly the dominant scaling strategy.
DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" (2024) -- Multi-head latent attention and DeepSeekMoE architecture. Frontier of efficient MoE design.
These papers answer the question: if you have a fixed compute budget, how should you allocate it between model size and data?
Kaplan et al., "Scaling Laws for Neural Language Models" (2020) -- Empirically establishes power-law relationships between compute, dataset size, model size, and loss. The paper that made "scaling" a research program.
Hoffmann et al., "Training Compute-Optimal Large Language Models" (2022) -- Chinchilla -- Revises the Kaplan scaling laws. Key finding: most large models are significantly undertrained. This paper directly caused the shift toward smaller models with more data.
Muennighoff et al., "Scaling Data-Constrained Language Models" (2023) -- What happens when you run out of unique training data? Covers data repetition, data quality vs. quantity, and code data.
Wei et al., "Emergent Abilities of Large Language Models" (2022) -- Documents capabilities that appear suddenly as models scale, rather than improving gradually. Controversial but important.
Schaeffer et al., "Are Emergent Abilities of Large Language Models a Mirage?" (2023) -- The counterargument: emergence may be an artifact of the metrics chosen. Read both this and the Wei et al. paper.
Olsson et al., "In-context Learning and Induction Heads" (2022) -- Anthropic -- A mechanistic account of how in-context learning actually works inside the transformer. One of the best examples of mechanistic interpretability producing genuine insight.
Garg et al., "What Can Transformers Learn In-Context? A Case Study of Simple Function Classes" (2022) -- Formalizes in-context learning as implicit Bayesian inference.
Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022) -- Showed that prompting the model to "think step by step" dramatically improves performance on reasoning tasks.
Kojima et al., "Large Language Models are Zero-Shot Reasoners" (2022) -- The "Let's think step by step" paper. Shows that chain-of-thought works even without few-shot examples.
Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (2023) -- Sample multiple reasoning paths and take the majority vote.
Yao et al., "Tree of Thoughts: Deliberate Problem Solving with Large Language Models" (2023) -- Extends chain-of-thought to tree search over reasoning paths.
Zhou et al., "Large Language Models Are Human-Level Prompt Engineers" (2023) -- APE -- Automated prompt optimization. Reframes prompt engineering from art to optimization problem.
Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" (2023) -- Models learn to call APIs by self-supervising on which tool calls improve predictions. Foundational paper for tool-augmented LLMs.
Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (2023) -- Interleaves reasoning traces with actions. The reasoning-action loop that most agent frameworks are built on.
Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning" (2023) -- Agents that reflect on their failures and improve.
Anthropic, "Computer Use" documentation and research (2024-2025) -- Models that can operate computer interfaces directly. Important for understanding where tool use is heading.
Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (2021) -- CLIP -- Contrastive learning between images and text. The foundation for connecting vision and language.
Alayrac et al., "Flamingo: a Visual Language Model for Few-Shot Learning" (2022) -- Interleaves visual and textual inputs in a frozen LLM. Important architecture for understanding how vision gets integrated into language models.
Liu et al., "Visual Instruction Tuning" (2023) -- LLaVA -- Simple and effective approach to building vision-language models.
OpenAI, "GPT-4V(ision) System Card" (2023) -- Documents capabilities and limitations of multimodal frontier models.
Team et al., "Gemini: A Family of Highly Capable Multimodal Models" (2024) -- Natively multimodal from pre-training. Represents the direction the field is moving.
Understanding how models are measured is as important as understanding how they work.
Liang et al., "Holistic Evaluation of Language Models" (2022) -- HELM -- Comprehensive evaluation framework covering accuracy, calibration, robustness, fairness, efficiency, and more.
Hendrycks et al., "Measuring Massive Multitask Language Understanding" (2021) -- MMLU -- 57-subject multiple choice benchmark. Widely used, widely criticized, still the default reference point.
Cobbe et al., "Training Verifiers to Solve Math Word Problems" (2021) -- GSM8K -- Grade school math benchmark. Clean testbed for measuring reasoning vs. pattern matching.
Chen et al., "Evaluating Large Language Models Trained on Code" (2021) -- HumanEval -- Code generation benchmark. Either the code runs or it does not.
Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023) -- Using LLMs to evaluate LLMs, plus the Elo-based Chatbot Arena.
Lilian Weng's Blog (lilianweng.github.io) -- Extraordinarily thorough survey posts. Start with "Prompt Engineering" and "LLM Powered Autonomous Agents."
Jay Alammar's Blog (jalammar.github.io) -- Visual explanations of architectures.
Sebastian Raschka's Blog and Newsletter -- Practical and rigorous. Good coverage of fine-tuning techniques, model comparisons, and new developments.
Anthropic Research Blog (anthropic.com/research) -- Interpretability, scaling, safety, and model behavior. Some of the most careful empirical work on LLM behavior.
OpenAI Research Blog (openai.com/research) -- New model releases, safety research, and capability demonstrations.
Google DeepMind Blog -- Gemini, scaling research, and multimodality.
Meta AI Blog (ai.meta.com/blog) -- LLaMA releases, open-weight model research, and infrastructure papers.
The Gradient (thegradient.pub) -- Long-form essays on ML research. Higher signal-to-noise ratio than most outlets.
Papers With Code (paperswithcode.com) -- Tracks state-of-the-art results by task.
Hugging Face NLP Course (huggingface.co/learn/nlp-course) -- Free course covering tokenizers, transformers, fine-tuning, and deployment. Where theory meets practice.
Hugging Face Transformers Documentation -- The reference for actually using transformer models. Read the conceptual guides, not just the API docs.
Andrej Karpathy, "nanoGPT" (GitHub repository) -- A minimal, readable GPT implementation. Train a small model on your own data.
Anthropic's Prompt Engineering Guide (docs.anthropic.com) -- Practical prompting techniques with worked examples.
For someone starting from zero:
After completing this sequence, you will be able to read current LLM research papers without needing to stop and look up every third concept.