Reinforcement Learning

Course manual 2022/2023

Course content

Reinforcement learning is a general framework studying sequential decision-making problems. In such problems, at every time step an action must be chosen to optimize long-term performance. This is a very wide class of problems that includes robotic control, game playing, but also human and animal behavior. Reinforcement learning methods can be applied when no training labels for the optimal action are available, and good actions have to be discovered through trial and error.

In this course, we will discuss properties of reinforcement learning problems and algorithms to solve them. In particular, we will look at

• Solving problems with discrete action sets using so-called value-based methods. We will cover approximate methods that can be employed in problems with large state spaces.
• Solving problems with continuous actions using approximate policy-based methods. These methods have application in, among others, robotics and control.
• The use of multi-layer neural networks as function approximators in reinforcement learning algorithms, yielding so-called deep reinforcement learning methods
• We will also briefly cover advanced & frontier topics in reinforcement learning.

Study materials

Literature

• Reinforcement Learning: An Introduction. R. S. Sutton & A. G. Barto
  Second Edition. Available: http://incompleteideas.net/book/RLbook2020.pdf. We will cover or partially cover chapters 1-11, 13, 16, and 17.

• A Survey on Policy Search for Robotics. M. P. Deisenroth, G. Neumann, J. Peters. We will cover this survey partially. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.436.44&rep=rep1&type=pdf

• We will study new developments in the field of RL through recent publicly available papers. Links will be distributed through the class website.

Objectives

• The student is able to describe the main algorithms in Monte Carlo, temporal difference, and model-based and policy-based methods
• The student is able to describe the main differences between on&off policy learning, TD&Monte Carlo methods, value-based and policy-based methods, tabular and approximate methods; and categorise reinforcement learning methods according to these properties
• The student is able to implement reinforcement learning algorithm; and able to analyse its performance (or lack thereof) in a given environment. The student is able to apply the learned update rule on given data sets.
• The student is able to compare reinforcement learning algorithms and point out their main advantages and disadvantages. The student is able to choose which reinforcement learning algorithm to select based on characteristics of the given environment
• The student is able to critically evaluate reinforcement learning experiments and point out their strong points and weak points.

Teaching methods

• Lecture
• Seminar

Learning activities

Attendance

This programme does not have requirements concerning attendance (OER part B).

Additional requirements for this course:

Attendance to lectures and tutorial sessions is strongly encouraged but not required. 

Assessment

A resit is possible for the exam only.

A result of at least 5.0 on the exam is necessary to pass the course. Of course, the weighted average of assignments and exam needs to be at least 5.5 to pass the course.

The final grade is a weighted average of assignments and the exam as follows: 

• Exam 65%
• 5 homeworks, 4% each
• 5 coding assignments, 2% each
• Reproducibility report, 5%

Assignments handed in after the assignment deadline without permission will not be graded. Limited extensions will only be given in exceptional circumstances. 

Inspection of assessed work

An announcement will be made on canvas for inspecting exam grades. For inspecting assignment grades, ask your TA after the grade is announced.

Assignments

Both graded and ungraded assignments are provided. Only the graded assignments should be handed in. They are clearly marked as 'homework'. 

The answers to ungraded assignments will be provided one week after the assignment was scheduled so that students can check their own work. 

Homework assignments, coding assignments and the reproducibility report can be done in groups of 2 or individually. Feedback will be given in Canvas and additional feedback can be given on request by the TA during tutorial sessions. 

Fraud and plagiarism

The 'Regulations governing fraud and plagiarism for UvA students' applies to this course. This will be monitored carefully. Upon suspicion of fraud or plagiarism the Examinations Board of the programme will be informed. For the 'Regulations governing fraud and plagiarism for UvA students' see: www.student.uva.nl

Course structure

Timetable

The schedule for this course is published on DataNose.

Contact information

Coordinator

• dr. H.C. van Hoof