#LLM Pre-Training: A Reading Curriculum

A structured reading list for going from zero to research-level understanding of large language model pre-training. Each phase builds on the last.

#Phase 0: Prerequisites

Before touching pre-training literature, you need working fluency in a few areas. If you already have these, skip ahead.

Linear Algebra and Calculus

• Gilbert Strang, Introduction to Linear Algebra (chapters 1-7) -- matrix operations, eigenvalues, SVD. You will not understand attention without this.
• 3Blue1Brown, Essence of Linear Algebra (YouTube series) -- geometric intuition for the above.

Probability and Information Theory

• Chapter 3 of Goodfellow, Bengio, Courville, Deep Learning (freely available at deeplearningbook.org) -- entropy, cross-entropy, KL divergence. These are the loss functions you will be staring at.

Deep Learning Fundamentals

• Andrej Karpathy, Neural Networks: Zero to Hero (YouTube playlist, 2022-2023) -- builds neural nets from scratch in Python. The single best on-ramp. Watch all of them.
• Goodfellow et al., Deep Learning, chapters 6-8 -- feedforward networks, regularization, optimization. Reference as needed.

PyTorch

• Official PyTorch tutorials (pytorch.org/tutorials) -- tensor operations, autograd, nn.Module. You need to be comfortable reading and writing model code.

#Phase 1: Foundations

The goal here is to understand the transformer architecture, the basic training loop, and the key ideas that make modern pre-training work.

#1.1 The Transformer

• Vaswani et al., "Attention Is All You Need" (2017) -- the paper that started everything. Read it carefully. Understand multi-head attention, positional encoding, encoder-decoder structure.
• Jay Alammar, "The Illustrated Transformer" (blog post, 2018) -- visual walkthrough of the above paper. Read this alongside or immediately after.
• Andrej Karpathy, "Let's build GPT: from scratch, in code, spelled out" (YouTube, 2023) -- builds a small GPT from scratch. Type along.

#1.2 Language Modeling Objectives

• Radford et al., "Improving Language Understanding by Generative Pre-Training" (GPT-1, 2018) -- introduces the idea of pre-training a causal language model then fine-tuning.
• Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers" (2018) -- masked language modeling (MLM) and next sentence prediction. Understand why bidirectional context matters and what trade-offs it introduces.
• Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (T5, 2019) -- systematic comparison of pre-training objectives. Section 3 is a goldmine for understanding design choices.
• Lewis et al., "BART: Denoising Sequence-to-Sequence Pre-training" (2019) -- denoising autoencoder approach to pre-training.

#1.3 Tokenization

• Sennrich, Haddow, Birch, "Neural Machine Translation of Rare Words with Subword Units" (2016) -- introduces Byte Pair Encoding (BPE) for NLP.
• Kudo and Richardson, "SentencePiece: A simple and language independent subword tokenizer" (2018) -- unigram language model tokenization and the SentencePiece library.
• Kudo, "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates" (2018) -- why tokenization is not deterministic and why that can help.
• Hugging Face Tokenizers documentation -- practical reference for BPE, WordPiece, Unigram, and Byte-level BPE.

#1.4 First Hands-On

• Karpathy, nanoGPT (github.com/karpathy/nanoGPT) -- minimal GPT training code. Read it end to end. Train a small model on Shakespeare or OpenWebText. This is not optional.
• Karpathy, minbpe (github.com/karpathy/minbpe) -- minimal BPE tokenizer implementation. Read alongside the BPE paper.

#Phase 2: Intermediate

Now you understand the basic loop: collect data, tokenize, define architecture, define objective, train. This phase is about understanding the engineering and scientific decisions that separate a toy model from a real one.

#2.1 Scaling and Landmark Models

• Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019) -- demonstrates that scale improves zero-shot performance. Introduces WebText dataset curation.
• Brown et al., "Language Models are Few-Shot Learners" (GPT-3, 2020) -- 175B parameters, in-context learning, few-shot prompting. Sections 2 (approach), 6 (broader impacts), and Appendix B (data) are essential.
• Kaplan et al., "Scaling Laws for Neural Language Models" (2020) -- empirical power laws relating compute, data, and parameters to loss. Read it thoroughly.
• Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla, 2022) -- overturns the Kaplan scaling laws. The "Chinchilla-optimal" ratio changed how everyone budgets training. Critical paper.

#2.2 Architecture Decisions

• Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021) -- Rotary Position Embedding (RoPE). Used by LLaMA, Mistral, and most modern open models.
• Press, Smith, Lewis, "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation" (ALiBi, 2021) -- alternative to learned or sinusoidal position embeddings. Used by BLOOM and MPT.
• Shazeer, "GLU Variants Improve Transformer" (2020) -- SwiGLU and other gated linear units for the FFN layer. SwiGLU is now standard in LLaMA-family models.
• Zhang and Sennrich, "Root Mean Square Layer Normalization" (RMSNorm, 2019) -- simpler alternative to LayerNorm. Used by LLaMA and successors.
• Shazeer, "Fast Transformer Decoding: One Write-Head is All You Need" (2019) -- multi-query attention (MQA). Reduces KV cache memory at inference time.
• Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023) -- grouped-query attention, the middle ground between MHA and MQA.

#2.3 Optimization

• Kingma and Ba, "Adam: A Method for Stochastic Optimization" (2014) -- the Adam optimizer. Understand the first and second moment estimates.
• Loshchilov and Hutter, "Decoupled Weight Decay Regularization" (AdamW, 2017) -- fixes weight decay in Adam. This is what modern training actually uses.
• Loshchilov and Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts" (2016) -- cosine annealing learning rate schedule. Standard in pre-training.
• Micikevicius et al., "Mixed Precision Training" (2017) -- training in FP16 with loss scaling. Essential for fitting large models in GPU memory.
• You et al., "Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes" (LAMB, 2019) -- layer-wise adaptive learning rates for large-batch training.

#2.4 Pre-Training Data

• Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Language Modeling" (2020) -- Eleuther AI's curated pre-training dataset. Read for the philosophy of data mixing.
• Penedo et al., "The RefinedWeb Dataset for Falcon LLM" (2023) -- demonstrates that heavily filtered web data alone can match curated multi-source datasets.
• Together AI, "RedPajama: An Open Dataset for Training Large Language Models" (2023) -- open reproduction of the LLaMA training data.
• Penedo et al., "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale" (2024) -- HuggingFace's large-scale filtered web dataset. State of the art in open data curation.
• Lee et al., "Deduplicating Training Data Makes Language Models Better" (2021) -- near-duplicate detection at scale. Deduplication meaningfully improves model quality.
• Wenzek et al., "CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data" (2019) -- perplexity-based filtering of Common Crawl.

#2.5 Key Open Model Papers

These papers are valuable because they disclose training details that closed labs do not.

• Touvron et al., "LLaMA: Open and Efficient Foundation Language Models" (2023) -- training recipe, architecture choices (RoPE, SwiGLU, RMSNorm, no bias), data mixture. Read sections 2-3 carefully.
• Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models" (2023) -- 7B to 70B, grouped-query attention at 70B, longer context.
• Biderman et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling" (2023) -- Eleuther AI's controlled experiment suite. Invaluable for studying training dynamics.

#Phase 3: Advanced

At this point you understand how pre-training works. This phase is about understanding the frontier.

#3.1 Distributed Training

• Li et al., "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel" (2023) -- Facebook's approach to sharding model parameters across GPUs.
• Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (2019) -- DeepSpeed's ZeRO optimizer. Understand the three stages of partitioning.
• Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" (2019) -- tensor parallelism for transformers.
• Narayanan et al., "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM" (2021) -- combines data, tensor, and pipeline parallelism (3D parallelism).
• Huang et al., "GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism" (2019) -- pipeline parallelism with micro-batches.
• DeepSpeed documentation (deepspeed.ai) -- reference for ZeRO stages, mixed precision, gradient checkpointing.

#3.2 Scaling Laws (Advanced)

• Clark et al., "Unified Scaling Laws for Routed Language Models" (2022) -- scaling laws for mixture-of-experts models.
• Muennighoff et al., "Scaling Data-Constrained Language Models" (2023) -- what happens when you run out of unique data?
• DeepSeek-AI, "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism" (2024) -- DeepSeek's own scaling law analysis.

#3.3 Frontier Pre-Training Papers

• Groeneveld et al., "OLMo: Accelerating the Science of Language Models" (2024) -- Allen AI's fully open model: code, data (Dolma), weights, training logs, evaluation. The most reproducible large-scale pre-training effort.
• Meta, "The Llama 3 Herd of Models" (2024) -- technical report for Llama 3 (8B, 70B, 405B). Extremely detailed on data pipeline, training recipe, and scaling. Essential reading.
• DeepSeek-AI, "DeepSeek-V2" (2024) -- multi-head latent attention (MLA) and DeepSeekMoE architecture. Novel attention compression for efficient inference.
• DeepSeek-AI, "DeepSeek-V3 Technical Report" (2024) -- 671B MoE trained on 14.8T tokens. FP8 training, load-balancing without auxiliary losses, multi-token prediction. One of the most cost-efficient frontier models.
• Jiang et al., "Mistral 7B" (2023) -- sliding window attention, grouped-query attention, efficient 7B model.
• Jiang et al., "Mixtral of Experts" (2024) -- sparse mixture of experts with 8 experts, 2 active.

#3.4 Data Quality and Filtering (Advanced)

• Marion et al., "When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale" (2023) -- perplexity-based data pruning.
• Xie et al., "DoReMi: Optimizing Data Mixtures for Language Modeling" (2023) -- learns optimal domain weights using a small proxy model.
• Longpre et al., "A Pretrainer's Guide to Training Data" (2024) -- comprehensive survey of pre-training data decisions.

#3.5 Curriculum Learning and Training Dynamics

• Bengio et al., "Curriculum Learning" (2009) -- the original curriculum learning paper.
• Chen et al., "Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models" (2023) -- ordering pre-training data by skill acquisition.
• Ibrahim et al., "Simple and Scalable Strategies to Continually Pre-train Large Language Models" (2024) -- practical techniques for continued pre-training without catastrophic forgetting.

#3.6 Multi-Token Prediction and Alternative Objectives

• Gloeckle et al., "Better & Faster Large Language Models via Multi-token Prediction" (Meta, 2024) -- predicting multiple future tokens simultaneously. Improves sample efficiency and downstream performance. Used in DeepSeek-V3.

#3.7 Infrastructure

• NVIDIA, "NVIDIA DGX SuperPOD" documentation -- reference architecture for large-scale GPU clusters.
• Patterson et al., "Carbon Emissions and Large Neural Network Training" (2021) -- the environmental and hardware cost of training.
• Luccioni et al., "Power Hungry Processing: Watts Driving the Cost of AI Deployment?" (2023) -- energy costs of training and inference.

#Phase 4: Research Practice

#4.1 Eleuther AI Ecosystem

Eleuther AI has produced the most important open infrastructure for pre-training research.

• GPT-NeoX (github.com/EleutherAI/gpt-neox) -- distributed training library.
• lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness) -- standard evaluation framework.
• The Pile (pile.eleuther.ai) -- the dataset. Read the datasheet.

#4.2 Other Practical Resources

• Hugging Face, The Large Language Model Course (github.com/huggingface/llm-course) -- structured course covering pre-training, evaluation, and deployment.
• Hugging Face nanotron (github.com/huggingface/nanotron) -- 3D parallelism training library.
• LitGPT (github.com/Lightning-AI/litgpt) -- clean implementation of multiple LLM architectures.
• Stas Bekman, "The Technology Behind BLOOM Training" (huggingface.co blog, 2022) -- detailed engineering post-mortem of training a 176B model. Extremely practical.
• Stas Bekman, "Machine Learning Engineering Open Book" (github.com/stas00/ml-engineering) -- comprehensive open reference on the engineering side.

#4.3 Surveys and Meta-Resources

• Zhao et al., "A Survey of Large Language Models" (2023, regularly updated) -- broad survey. Use as a map.
• Minaee et al., "Large Language Models: A Survey" (2024) -- more focused on architecture and training methodology.

#Suggested Reading Order

If you want a linear path through this material:

1. Karpathy's Neural Networks: Zero to Hero (video series)
2. "Attention Is All You Need" + Jay Alammar's illustrated version
3. Karpathy's "Let's build GPT" + nanoGPT hands-on
4. GPT-1 paper (short, sets the frame)
5. BPE paper + minbpe code
6. BERT paper (for contrast with causal LM)
7. T5 paper sections 2-3 (training objective comparison)
8. GPT-3 paper
9. Kaplan scaling laws
10. Chinchilla paper
11. LLaMA paper
12. AdamW paper + mixed precision paper
13. The Pile paper + deduplication paper
14. Pythia paper
15. ZeRO paper + Megatron-LM paper
16. Llama 3 technical report
17. OLMo paper
18. FineWeb paper
19. DeepSeek-V3 technical report
20. Stas Bekman's ML Engineering book (ongoing reference)

#Notes on Approach

Read papers with code open. Whenever possible, find the implementation. nanoGPT for the basic loop, Pythia/OLMo for full-scale training code, nanotron for distributed training.

Reproduce small-scale results. You do not need a cluster to learn pre-training. Train a 124M parameter model on a single GPU. The dynamics are the same.

Track the data story. The evolution from WebText to The Pile to RefinedWeb to FineWeb is as important as the architecture story. Data quality is the primary lever at scale.

Understand the economics. Scaling laws determine how labs allocate millions of dollars in compute. If you cannot reason about compute-optimal training, you cannot contribute to pre-training research.