Overview
- Lectures
- Tue & Thu 2:55-4:10pm, Upson Hall 142
- Instructor
- Zhiru Zhang
- Office hours
- Thursday 5:00-6:00pm, Online
Course Texts
- Lecture slides posted in the course schedule
- Austin et al., How to Scale Your Model, Google DeepMind, online, 2025. (Scaling Book)
Fall 2026
Course Schedule
Lecture
Deadline
Break
Required reading
Supplementary reading
Week 1
Tue 08/25 – Introduction
- Lecture01 Slides
- Readings
Thu 08/27 – Basics of Neural Networks
- Lecture02 Slides
- Readings
- Stanford CS231n, Neural Networks Part 1.
- J. Kun, The Perceptron, and All the Things it Can't Perceive.
Week 2
Tue 09/01 – Hardware Specialization, Part 1
- Lecture03 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Thu 09/03 – Hardware Specialization, Part 2
- Lecture04 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Week 3
Tue 09/08 – Roofline Analysis, Tiling
- Lecture05 Slides
- Readings
- Scaling Book Ch. 1: All About Rooflines.
- Williams et al., Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 2009.
- S. Boehm, Fast Multidimensional Matrix Multiplication on CPU from Scratch.
Thu 09/10 – Tutorial on Lab Setup
Fri 09/11 – HW 1 Due
Week 4
Mon 09/14 – Lab 0 Due: Introducing MiniTPU
Tue 09/15 – Systolic Arrays
- Lecture07 Slides
- Readings
- Scaling Book Ch. 2: Appendix B: How does a systolic array work?
- H.T. Kung, Why Systolic Architecture?, 1982.
Thu 09/17 – TPUs
- Lecture08 Slides
- Readings
- Scaling Book Ch. 2: How to Think About TPUs.
- Jouppi et al., A Domain-Specific Supercomputer for Training Deep Neural Networks, CACM 2020.
Week 5
Tue 09/22 – Sharding
- Lecture09 Slides
- Readings
- Scaling Book Ch. 3: Sharded Matrices and How to Multiply Them.
Wed 09/23 – Lab 1 Due
Thu 09/24 – GPUs, Part 1
- Lecture10 Slides
- Readings
- Scaling Book Ch. 12: How to Think About GPUs.
- Y. Kreinin, SIMD < SIMT < SMT: Parallelism in NVIDIA GPUs.
Week 6
Mon 09/28 – HW 2 Due
Tue 09/29 – GPUs, Part 2
- Lecture11 Slides
- Readings
- Scaling Book Ch. 12: How to Think About GPUs.
- CMU, Modern GPU Programming For MLSys: Data Layout and Its Notation.
Thu 10/01 – Guest Lecture by Dr. Stephen Neuendorffer (AMD)
Week 7
Tue 10/06 – Tutorial on Kernel Programming with Triton
- Lecture13 Slides
- Readings
- S. Rush, Triton Puzzles.
Thu 10/08 – In-Class Prelim
- Open Book, Open Notes
Week 8
Tue 10/13 – Fall Break
Thu 10/15 – LLMs, Part 1
- Lecture14 Slides
- Readings
- 3Blue1Brown: 1. Transformers, the tech behind LLMs, 2. Attention in transformers, step-by-step.
- Scaling Book Ch. 4: All the Transformer Math You Need to Know.
Week 9
Mon 10/19 – Lab 2 Due
Tue 10/20 – Guest Lecture by Dr. Shreesha Srinath (Cerebras)
Thu 10/22 – LLMs, Part 2
- Lecture16 Slides
- Readings
- Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022.
- Scaling Book Ch. 7: All About Transformer Inference.
Week 10
Tue 10/27 – Model-Level Optimization
- Lecture17 Slides
- Readings
- M. Grootendorst, A Visual Guide to Quantization.
- Welch Labs, How DeepSeek Rewrote the Transformer [MLA].
Thu 10/29 – Guest Lecture by Steve Yalovitser and Shivika Singh (AWS)
Fri 10/30 – HW 3 Due
Week 11
Tue 11/03 – LLM Serving
- Lecture19 Slides
- Readings
- Scaling Book Ch. 7: All About Transformer Inference.
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.
- Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, NeurIPS 2024.