# LLM Pre-Training: A Reading Curriculum A structured reading list for going from zero to research-level understanding of large language model pre-training. Each phase builds on the last. --- ## Phase 0: Prerequisites Before touching pre-training literature, you need working fluency in a few areas. If you already have these, skip ahead. **Linear Algebra and Calculus** - Gilbert Strang, *Introduction to Linear Algebra* (chapters 1-7) -- matrix operations, eigenvalues, SVD. You will not understand attention without this. - 3Blue1Brown, *Essence of Linear Algebra* (YouTube series) -- geometric intuition for the above. **Probability and Information Theory** - Chapter 3 of Goodfellow, Bengio, Courville, *Deep Learning* (freely available at deeplearningbook.org) -- entropy, cross-entropy, KL divergence. These are the loss functions you will be staring at. **Deep Learning Fundamentals** - Andrej Karpathy, *Neural Networks: Zero to Hero* (YouTube playlist, 2022-2023) -- builds neural nets from scratch in Python. The single best on-ramp. Watch all of them. - Goodfellow et al., *Deep Learning*, chapters 6-8 -- feedforward networks, regularization, optimization. Reference as needed. **PyTorch** - Official PyTorch tutorials (pytorch.org/tutorials) -- tensor operations, autograd, nn.Module. You need to be comfortable reading and writing model code. --- ## Phase 1: Foundations The goal here is to understand the transformer architecture, the basic training loop, and the key ideas that make modern pre-training work. ### 1.1 The Transformer - **Vaswani et al., "Attention Is All You Need" (2017)** -- the paper that started everything. Read it carefully. Understand multi-head attention, positional encoding, encoder-decoder structure. - **Jay Alammar, "The Illustrated Transformer" (blog post, 2018)** -- visual walkthrough of the above paper. Read this alongside or immediately after. - **Andrej Karpathy, "Let's build GPT: from scratch, in code, spelled out" (YouTube, 2023)** -- builds a small GPT from scratch. Type along. ### 1.2 Language Modeling Objectives - **Radford et al., "Improving Language Understanding by Generative Pre-Training" (GPT-1, 2018)** -- introduces the idea of pre-training a causal language model then fine-tuning. - **Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers" (2018)** -- masked language modeling (MLM) and next sentence prediction. Understand why bidirectional context matters and what trade-offs it introduces. - **Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (T5, 2019)** -- systematic comparison of pre-training objectives. Section 3 is a goldmine for understanding design choices. - **Lewis et al., "BART: Denoising Sequence-to-Sequence Pre-training" (2019)** -- denoising autoencoder approach to pre-training. ### 1.3 Tokenization - **Sennrich, Haddow, Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016)** -- introduces Byte Pair Encoding (BPE) for NLP. - **Kudo and Richardson, "SentencePiece: A simple and language independent subword tokenizer" (2018)** -- unigram language model tokenization and the SentencePiece library. - **Kudo, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates" (2018)** -- why tokenization is not deterministic and why that can help. - **Hugging Face Tokenizers documentation** -- practical reference for BPE, WordPiece, Unigram, and Byte-level BPE. ### 1.4 First Hands-On - **Karpathy, nanoGPT (github.com/karpathy/nanoGPT)** -- minimal GPT training code. Read it end to end. Train a small model on Shakespeare or OpenWebText. This is not optional. - **Karpathy, minbpe (github.com/karpathy/minbpe)** -- minimal BPE tokenizer implementation. Read alongside the BPE paper. --- ## Phase 2: Intermediate Now you understand the basic loop: collect data, tokenize, define architecture, define objective, train. This phase is about understanding the engineering and scientific decisions that separate a toy model from a real one. ### 2.1 Scaling and Landmark Models - **Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019)** -- demonstrates that scale improves zero-shot performance. Introduces WebText dataset curation. - **Brown et al., "Language Models are Few-Shot Learners" (GPT-3, 2020)** -- 175B parameters, in-context learning, few-shot prompting. Sections 2 (approach), 6 (broader impacts), and Appendix B (data) are essential. - **Kaplan et al., "Scaling Laws for Neural Language Models" (2020)** -- empirical power laws relating compute, data, and parameters to loss. Read it thoroughly. - **Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla, 2022)** -- overturns the Kaplan scaling laws. The "Chinchilla-optimal" ratio changed how everyone budgets training. Critical paper. ### 2.2 Architecture Decisions - **Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021)** -- Rotary Position Embedding (RoPE). Used by LLaMA, Mistral, and most modern open models. - **Press, Smith, Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ALiBi, 2021)** -- alternative to learned or sinusoidal position embeddings. Used by BLOOM and MPT. - **Shazeer, "GLU Variants Improve Transformer" (2020)** -- SwiGLU and other gated linear units for the FFN layer. SwiGLU is now standard in LLaMA-family models. - **Zhang and Sennrich, "Root Mean Square Layer Normalization" (RMSNorm, 2019)** -- simpler alternative to LayerNorm. Used by LLaMA and successors. - **Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019)** -- multi-query attention (MQA). Reduces KV cache memory at inference time. - **Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023)** -- grouped-query attention, the middle ground between MHA and MQA. ### 2.3 Optimization - **Kingma and Ba, "Adam: A Method for Stochastic Optimization" (2014)** -- the Adam optimizer. Understand the first and second moment estimates. - **Loshchilov and Hutter, "Decoupled Weight Decay Regularization" (AdamW, 2017)** -- fixes weight decay in Adam. This is what modern training actually uses. - **Loshchilov and Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts" (2016)** -- cosine annealing learning rate schedule. Standard in pre-training. - **Micikevicius et al., "Mixed Precision Training" (2017)** -- training in FP16 with loss scaling. Essential for fitting large models in GPU memory. - **You et al., "Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes" (LAMB, 2019)** -- layer-wise adaptive learning rates for large-batch training. ### 2.4 Pre-Training Data - **Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Language Modeling" (2020)** -- Eleuther AI's curated pre-training dataset. Read for the philosophy of data mixing. - **Penedo et al., "The RefinedWeb Dataset for Falcon LLM" (2023)** -- demonstrates that heavily filtered web data alone can match curated multi-source datasets. - **Together AI, "RedPajama: An Open Dataset for Training Large Language Models" (2023)** -- open reproduction of the LLaMA training data. - **Penedo et al., "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale" (2024)** -- HuggingFace's large-scale filtered web dataset. State of the art in open data curation. - **Lee et al., "Deduplicating Training Data Makes Language Models Better" (2021)** -- near-duplicate detection at scale. Deduplication meaningfully improves model quality. - **Wenzek et al., "CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data" (2019)** -- perplexity-based filtering of Common Crawl. ### 2.5 Key Open Model Papers These papers are valuable because they disclose training details that closed labs do not. - **Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (2023)** -- training recipe, architecture choices (RoPE, SwiGLU, RMSNorm, no bias), data mixture. Read sections 2-3 carefully. - **Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023)** -- 7B to 70B, grouped-query attention at 70B, longer context. - **Biderman et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling" (2023)** -- Eleuther AI's controlled experiment suite. Invaluable for studying training dynamics. --- ## Phase 3: Advanced At this point you understand how pre-training works. This phase is about understanding the frontier. ### 3.1 Distributed Training - **Li et al., "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel" (2023)** -- Facebook's approach to sharding model parameters across GPUs. - **Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2019)** -- DeepSpeed's ZeRO optimizer. Understand the three stages of partitioning. - **Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (2019)** -- tensor parallelism for transformers. - **Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM" (2021)** -- combines data, tensor, and pipeline parallelism (3D parallelism). - **Huang et al., "GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism" (2019)** -- pipeline parallelism with micro-batches. - **DeepSpeed documentation (deepspeed.ai)** -- reference for ZeRO stages, mixed precision, gradient checkpointing. ### 3.2 Scaling Laws (Advanced) - **Clark et al., "Unified Scaling Laws for Routed Language Models" (2022)** -- scaling laws for mixture-of-experts models. - **Muennighoff et al., "Scaling Data-Constrained Language Models" (2023)** -- what happens when you run out of unique data? - **DeepSeek-AI, "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism" (2024)** -- DeepSeek's own scaling law analysis. ### 3.3 Frontier Pre-Training Papers - **Groeneveld et al., "OLMo: Accelerating the Science of Language Models" (2024)** -- Allen AI's fully open model: code, data (Dolma), weights, training logs, evaluation. The most reproducible large-scale pre-training effort. - **Meta, "The Llama 3 Herd of Models" (2024)** -- technical report for Llama 3 (8B, 70B, 405B). Extremely detailed on data pipeline, training recipe, and scaling. Essential reading. - **DeepSeek-AI, "DeepSeek-V2" (2024)** -- multi-head latent attention (MLA) and DeepSeekMoE architecture. Novel attention compression for efficient inference. - **DeepSeek-AI, "DeepSeek-V3 Technical Report" (2024)** -- 671B MoE trained on 14.8T tokens. FP8 training, load-balancing without auxiliary losses, multi-token prediction. One of the most cost-efficient frontier models. - **Jiang et al., "Mistral 7B" (2023)** -- sliding window attention, grouped-query attention, efficient 7B model. - **Jiang et al., "Mixtral of Experts" (2024)** -- sparse mixture of experts with 8 experts, 2 active. ### 3.4 Data Quality and Filtering (Advanced) - **Marion et al., "When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale" (2023)** -- perplexity-based data pruning. - **Xie et al., "DoReMi: Optimizing Data Mixtures for Language Modeling" (2023)** -- learns optimal domain weights using a small proxy model. - **Longpre et al., "A Pretrainer's Guide to Training Data" (2024)** -- comprehensive survey of pre-training data decisions. ### 3.5 Curriculum Learning and Training Dynamics - **Bengio et al., "Curriculum Learning" (2009)** -- the original curriculum learning paper. - **Chen et al., "Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models" (2023)** -- ordering pre-training data by skill acquisition. - **Ibrahim et al., "Simple and Scalable Strategies to Continually Pre-train Large Language Models" (2024)** -- practical techniques for continued pre-training without catastrophic forgetting. ### 3.6 Multi-Token Prediction and Alternative Objectives - **Gloeckle et al., "Better & Faster Large Language Models via Multi-token Prediction" (Meta, 2024)** -- predicting multiple future tokens simultaneously. Improves sample efficiency and downstream performance. Used in DeepSeek-V3. ### 3.7 Infrastructure - **NVIDIA, "NVIDIA DGX SuperPOD" documentation** -- reference architecture for large-scale GPU clusters. - **Patterson et al., "Carbon Emissions and Large Neural Network Training" (2021)** -- the environmental and hardware cost of training. - **Luccioni et al., "Power Hungry Processing: Watts Driving the Cost of AI Deployment?" (2023)** -- energy costs of training and inference. --- ## Phase 4: Research Practice ### 4.1 Eleuther AI Ecosystem Eleuther AI has produced the most important open infrastructure for pre-training research. - **GPT-NeoX (github.com/EleutherAI/gpt-neox)** -- distributed training library. - **lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness)** -- standard evaluation framework. - **The Pile (pile.eleuther.ai)** -- the dataset. Read the datasheet. ### 4.2 Other Practical Resources - **Hugging Face, *The Large Language Model Course* (github.com/huggingface/llm-course)** -- structured course covering pre-training, evaluation, and deployment. - **Hugging Face nanotron (github.com/huggingface/nanotron)** -- 3D parallelism training library. - **LitGPT (github.com/Lightning-AI/litgpt)** -- clean implementation of multiple LLM architectures. - **Stas Bekman, "The Technology Behind BLOOM Training" (huggingface.co blog, 2022)** -- detailed engineering post-mortem of training a 176B model. Extremely practical. - **Stas Bekman, "Machine Learning Engineering Open Book" (github.com/stas00/ml-engineering)** -- comprehensive open reference on the engineering side. ### 4.3 Surveys and Meta-Resources - **Zhao et al., "A Survey of Large Language Models" (2023, regularly updated)** -- broad survey. Use as a map. - **Minaee et al., "Large Language Models: A Survey" (2024)** -- more focused on architecture and training methodology. --- ## Suggested Reading Order If you want a linear path through this material: 1. Karpathy's *Neural Networks: Zero to Hero* (video series) 2. "Attention Is All You Need" + Jay Alammar's illustrated version 3. Karpathy's "Let's build GPT" + nanoGPT hands-on 4. GPT-1 paper (short, sets the frame) 5. BPE paper + minbpe code 6. BERT paper (for contrast with causal LM) 7. T5 paper sections 2-3 (training objective comparison) 8. GPT-3 paper 9. Kaplan scaling laws 10. Chinchilla paper 11. LLaMA paper 12. AdamW paper + mixed precision paper 13. The Pile paper + deduplication paper 14. Pythia paper 15. ZeRO paper + Megatron-LM paper 16. Llama 3 technical report 17. OLMo paper 18. FineWeb paper 19. DeepSeek-V3 technical report 20. Stas Bekman's ML Engineering book (ongoing reference) --- ## Notes on Approach **Read papers with code open.** Whenever possible, find the implementation. nanoGPT for the basic loop, Pythia/OLMo for full-scale training code, nanotron for distributed training. **Reproduce small-scale results.** You do not need a cluster to learn pre-training. Train a 124M parameter model on a single GPU. The dynamics are the same. **Track the data story.** The evolution from WebText to The Pile to RefinedWeb to FineWeb is as important as the architecture story. Data quality is the primary lever at scale. **Understand the economics.** Scaling laws determine how labs allocate millions of dollars in compute. If you cannot reason about compute-optimal training, you cannot contribute to pre-training research.