CS 2824: Foundations of Reinforcement Learning

Staff

Instructors: Kianté Brantley and Sham Kakade

TFs: Lukas Fesser, Jaeyeon Kim, and Alex Meterez.

Lecture time: Tuesday/Thursday 12:45-2p

Office hours: By Appointment

Location: SEC LL2.224

Contact: Please communicate to the instructors and TFs only through the Ed account. Emails not sent to this list, with regards to the course, will not be responded to in a timely manner.

Announcements: Course announcements will be made via Canvas and Edstem. It is the students' responsibility to follow both.

Prerequisites

This is an advanced and theory-heavy course: there is no programming assignment and students are required to work on a theory-focused course project. Students need a strong grasp on Machine Learning, Probability and Statistics, Optimization, and Linear Algebra. For undergraduate and masters students enrollment: permission of instructor required through course petition.

Grading Policies

Assignments 60 Homework%, Project 30%, Reading 10%, (+Participation bonus 5%)

All homework will be mathematical in nature, focussing on the theory of RL and bandits; there will not be a programming component. The entire HW must be submitted in one single typed pdf document (not handwritten). HW0 is MANDATORY to pass to satisfactory level; it is to check your knowledge of the prerequisites in probability, statistics, and linear algebra.

Homework Rules: Homework must be done individually: each student must understand, write, and hand in their own answers. It is acceptable for students to discuss problems with each other; it is not acceptable for students to share answers and look at another students written answers. You must also indicate on each homework with whom you collaborated with and what online resources you used. You must attempt and submit all HW (even if it is for 0 credit) in order to pass the class.

Late days: Homeworks and Reading Assignments must be submitted by the posted due date. You are allowed up to 6 total LATE DAYs for the homeworks and reading assignments throughout the entire semester. These will be automatically deducted if your assignment is late. For example, any day in which an assignment is late by up to 24 hours, then one late day will be used. After your late days are used up, late penalties will be applied: any assignment turned in late will incur a reduction in score by 33% for each late day, so if an assignment is up to 24 hours late, it incurs a penalty of 33%. Else if it is up to 48 hours late, it incurs a penalty of 66%. And any longer, it will receive no credit. We will track all your late days and any deductions will be applied in computing the final grades. If you are unable to turn in HWs on time, aside from permitted days, then do not enroll in the course.

Regrading: If we made a mistake, you must let us know (in writing via Ed) within a week of when the HW was returned.

Participation/extra effort bonus: We encourage participation including asking/answering questions in lectures and ED discussion, and extra effort on reading the book chapters (e.g., proof reading additional chapters and sending back comments/feedback).

Reading Assignment

Reading assignments are meant to be completed actively and carefully. Student are responsible for reading the assigned readings in "Reinforcement Learning Theory and Algorithms" (ABJKS pdf link here) and engaging with the text to support learning. Note that LATE DAY POLICY also applies to the reading assignments. The readings are intended to help you develop a strong, working mastery of the material.

Students are encouraged to use ChatGPT (or another LLM tool) for all reading assignments. Harvard-enrolled students should have access to a university-sponsored account.

Course Project

Please see the course projects from ideas from an older version of the course page. Students will do project presentations during the last three lectures of the course. It is a course requirement that you be in attendance for all student presentations. See the dates below. Only the dates of 04/23/26 and 04/28/26 will be excused for ICLR, with instructor permission.

Diversity in STEM

While many academic disciplines have historically been dominated by one cross section of society, the study of and participation in STEM disciplines is a joy that the instructors hope that everyone can pursue, regardless of their socio-economic background, race, gender, etc. The instructors encourage students to both be mindful of these issues, and, in good faith, try to take steps to fix them. You are the next generation here.

Course Notes: RL Theory and Algorithms

The course will be largely based of the working draft of the book "Reinforcement Learning Theory and Algorithms". We will be updating the notes in ABJKS throughout the course of the term. If you find typos or errors, please let us know. We would appreciate it!

Tentative Dates (see Ed for announcements)

HW0: Due 01/30
HW1: Out 02/05, Due 02/16
HW2: Out 02/24, Due 03/13
HW3: Out 03/24, Due 04/08

Schedule (tentative)