Overview
- Lectures
- Tue & Thu 2:55-4:10pm, Upson Hall 142
- Instructor
- Zhiru Zhang
- Office hours
- Thursday 5:00-6:00pm, Online
Course Texts
- Lecture slides posted in the course schedule
- Austin et al., How to Scale Your Model, Google DeepMind, online, 2025. (Scaling Book)
Fall 2026
Course Schedule
Lecture
Deadline
Break
Required reading
Supplementary reading
Week 1
Tue 08/25 – Introduction
- Lecture01 Slides
- Readings
Thu 08/27 – Basics of Neural Networks
- Lecture02 Slides
- Readings
- Stanford CS231n, Neural Networks Part 1.
- J. Kun, The Perceptron, and All the Things it Can't Perceive.
Week 2
Tue 09/01 – Hardware Specialization, Part 1
- Lecture03 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Thu 09/03 – Hardware Specialization, Part 2
- Lecture04 Slides
- Readings
- Dally et al., Domain-Specific Hardware Accelerators, CACM 2020.
- Leiserson et al., There’s Plenty of Room at the Top: What will Drive Computer Performance after Moore’s Law?, Science 2020.
Week 3
Tue 09/08 – Roofline Analysis
- Lecture05 Slides
- Readings
- Scaling Book Part 1: All About Rooflines.
- Williams et al., Roofline: An Insightful Visual Performance Model for Multicore Architectures, CACM 2009.
Thu 09/10 – Tutorial on Lab Setup
Fri 09/11 – HW 1 Due
Week 4
Mon 09/14 – Lab 0 Due
Tue 09/15 – Systolic Arrays, TPUs
- Lecture07 Slides
- Scaling Book Part 2: How to Think About TPUs.
- Jouppi et al., A Domain-Specific Supercomputer for Training Deep Neural Networks, CACM 2020.
Thu 09/17 – Sharding
- Lecture08 Slides
- Readings
- Scaling Book Part 3: Sharded Matrices and How to Multiply Them.
Week 5
Mon 09/21 – Lab 1 Due
Tue 09/22 – GPUs, Part 1
- Lecture09 Slides
- Scaling Book Part 12: How to Think About GPUs.
Thu 09/24 – GPUs, Part 2
Week 6
Mon 09/28 – HW 2 Due
Tue 09/29 – LLMs, Part 1
- Lecture11 Slides
- Readings
- Scaling Book Part 4: All the Transformer Math You Need to Know.
- 3Blue1Brown: 1. Transformers, the tech behind LLMs, 2. Attention in transformers, step-by-step.
Thu 10/01 – Guest Lecture
Week 7
Tue 10/06 – Tutorial on Kernel Programming with Triton
Thu 10/08 – In-Class Prelim
- Open Book, Open Notes