Overview
- Lectures
- Tue & Thu 2:55-4:10pm, Upson Hall 142
- Instructor
- Zhiru Zhang
- Office hours
- Thursday 5:00-6:00pm, Online
Course Texts
- Lecture slides posted in the course schedule
- Austin et al., How to Scale Your Model, Google DeepMind, online, 2025. (Scaling Book)
Fall 2026
Course Schedule
Lecture
Deadline
Break
Required reading
Supplementary reading
Week 1
Tue 08/25 – Introduction
- Lecture01 Slides
- Readings
Thu 08/27 – Basics of Neural Networks
- Lecture02 Slides
- Readings
- Stanford CS231n, Neural Networks Part 1.
- J. Kun, The Perceptron, and All the Things it Can't Perceive.
Week 2
Tue 09/01 – Hardware Specialization, Part 1
- Lecture03 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Thu 09/03 – Hardware Specialization, Part 2
- Lecture04 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Week 3
Tue 09/08 – Roofline Analysis, Tiling
- Lecture05 Slides
- Readings
- Scaling Book Ch. 1: All About Rooflines.
- Williams et al., Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 2009.
- S. Boehm, Fast Multidimensional Matrix Multiplication on CPU from Scratch.
Thu 09/10 – Tutorial on Lab Setup
Fri 09/11 – HW 1 Due
Week 4
Mon 09/14 – Lab 0 Due: Introducing MiniTPU
Tue 09/15 – Systolic Arrays
- Lecture07 Slides
- Readings
- Scaling Book Ch. 2: Appendix B: How does a systolic array work?
- H.T. Kung, Why Systolic Architecture?, 1982.
Thu 09/17 – TPUs
- Lecture08 Slides
- Readings
- Scaling Book Ch. 2: How to Think About TPUs.
- Jouppi et al., A Domain-Specific Supercomputer for Training Deep Neural Networks, CACM 2020.
Week 5
Tue 09/22 – Sharding
- Lecture09 Slides
- Readings
- Scaling Book Ch. 3: Sharded Matrices and How to Multiply Them.
Wed 09/23 – Lab 1 Due
Thu 09/24 – GPUs, Part 1
- Lecture10 Slides
- Readings
- Scaling Book Ch. 12: How to Think About GPUs.
Week 6
Mon 09/28 – HW 2 Due
Tue 09/29 – GPUs, Part 2
Thu 10/01 – Guest Lecture
Week 7
Tue 10/06 – Tutorial on Kernel Programming with SPMD
Thu 10/08 – In-Class Prelim
- Open Book, Open Notes
Week 8
Tue 10/13 – Fall Break
Thu 10/15 – LLMs, Part 1
- Lecture14 Slides
- Readings
- Scaling Book Ch. 4: All the Transformer Math You Need to Know.
- 3Blue1Brown: 1. Transformers, the tech behind LLMs, 2. Attention in transformers, step-by-step.
Week 9
Mon 10/19 – Lab 2 Due
Tue 10/20 – Guest Lecture
Thu 10/22 – LLMs, Part 2
- Lecture16 Slides
- Readings
- Scaling Book Ch. 7: All About Transformer Inference.
Week 10
Mon 10/26 – HW 3 Due
Tue 10/27 – Guest Lecture
Thu 10/29 – Sparsity, Quantization
Week 11
Tue 11/03 – Project Guidelines
Thu 11/05 – LLM Serving
- Lecture20 Slides
- Readings
- Scaling Book Ch. 7: All About Transformer Inference.